Fast and flexible log normalization library
Fast and flexible log normalization library
When trying to normalize log messages via liblognorm and mmnormalize, you need to create a rulebase first. The rulebase is usually a representation of message formats. Due to the format of these rules, it is necessary to be cautious. Some messages and rule necessities could possibly cause confusion to the configuration interpreter. This mainly applies […]
When using log normalization, nothing is simple. But we have been asked something about a very common scenario. A log message has been sent to rsyslog. The message itself had no irregular characters. But, the message that should have been parsed by mmnormalize now has a leading space character. Basically, the message that should be […]
Using the mmnormalize module in rsyslog is a bit complicated at first. We want to describe in this article how to set up the basic components for using log normalization. In addition to that we will show how to configure these components so messages will be split into pieces of information. These pieces of information […]
-r = path to the rulebase -o = output format (Encoder) (just in V 0.1.0) -e = output format (Encoder) (since V 0.2.0) !!! -E = here you insert the fields that should be dispended (-E “host tag” -> that only dispend the host and the tag field) by default all fields will be dispended […]
To get a better overview of a rulebase you can create a graph that shows you the chain of normalization. At first you have to install an additional package called graphviz. Graphviz is a tool that creates such a graph with the help of a control file (created with the rulebase). Here you will find […]
A first example for a rulebase you can download at http://blog.gerhards.net/2010/11/log-normalization-first-results.html I will use an excerpt of that rulebase to show you the most common expressions. rule=:%date:date-rfc3164% %host:word% %tag:char-to:\x3a%: no longer listening on %ip:ipv4%#%port:number%’ That excerpt is a common rule. A rule contains different “parts”/properties, like the message you want to normalize (e.g. Host, IP, […]
Here you can find the first steps to use the pre-release of liblognorm. (Please note that the used operating system was Fedora 13.) At the moment there are two ways to install libognorm. You can install everything you need from git (below you can find all commands you need) or you can download it as […]