liblognorm

Blog

  • New Mailing List for Log Normalization

    Thankfully, the interest in log normalization and the related libraries liblognorm and libee has increased. Up until now, we have handled discussions on this topics via the rsyslog mailing list. As conversations increase, this may be come an unnecessary burden for those only interested in rsyslog. So we have created a new mailing list named lognorm. We used this somewhat generic name, as we intend to use it for both libraries. This saves me some overhead, and we strongly assume that anyone interested in liblognorm will also be interested in libee (but to a lesser extent in the reverse direction).

    Please subscribe to the new lists. Currently, it is a very exciting phase in log normalization development, so getting involved is a great way to shape things in the way you need it!

  • Available options for normalizer

    -r = path to the rulebase

    -o = output format (Encoder) (just in V 0.1.0)

    -e = output format (Encoder) (since V 0.2.0) !!!

    -E = here you insert the fields that should be dispended (-E “host tag” -> that only dispend the host and the tag field) by default all fields will be dispended

    -p = just the parsed messages will be dispensed (since V 0.2.0)

    -v = debug outout (-v is the normal debug mode; -vv is an expanded debug mode with more information)

    -d = dot file (Is used for creating a graph of the rulebase)

  • liblognorm 0.1.0 has been released

    Liblognorm is a event and log normalization library that is capable of real-time processing. It provides the capability to normalize events to a set of standard formats. It is most efficient when used together with almost unstructured data, as for example found in typical syslog messages. While liblognorm provides a service similar to other projects, it is unique in the way it works:

    a) As a library, it’s services can be used by any third-party application with ease. As such, other projects can benefit from liblognorm functionality without the user even knowing.

    b) Liblognorm is very fast, and especially much faster than regular-expression based solutions. This is possible because it uses a special data structure which (kind of) combines many regular expressions into a single, and thus faster, big one.

    This is the initial public release, targeted at early adopters. We will continue to enhance considerably, but the 0.1.0 version offers decent stability and features and so can actually be used.

    You can download it here.

  • liblognorm 0.1.0

    Download file name: liblognorm 0.1.0

    liblognorm 0.1.0
    md5sum: 72d90438b21e23805a0ab9312916782d

    Author: Rainer Gerhards (rgerhards@adiscon.com)
    Version: 0.1.0 File size: 0.307 MB

    Download this file now!

  • Rulebase

    A rulesbase is a collection of rules. The rulebase is the core of the normalizing process as it holds the information that is needed to transform logs into a common format.

    Upon execution of the normalizer, it will be transferred into a parse-tree.

  • Rule

    A rule is a a scheme that fits to a specific type of logfile from a specific device. It consists of multiple fields, which reflect the type of information. Many of these rules together build the rulebase.

  • log normalization with rsyslog

    We just wanted to give you a quick heads-up on our current development efforts: We have begun to work heavily on a message modfication module for rsyslog which will support liblognorm-style normalization inside rsyslog. In git there already is a branch “lognorm”, which we will hopefully complete and merge into master soon. It provides some very interesting shortcuts of pulling specific information out of syslog messages. We will probably promote it some more when it is available. IMHO it’s the coolest and potentially most valuable feature we have added in the past three years. Once we have enabled tags in liblognorm/libee, you can even very easily classify log messagesbased on their content.

  • Creating a graph of the rulebase

    To get a better overview of a rulebase you can create a graph that shows you the chain of normalization.

    At first you have to install an additional package called graphviz. Graphviz is a tool that creates such a graph with the help of a control file (created with the rulebase). Here you will find more informaton about graphviz.

    To install it you can use the package manager or the yum command.

    $ sudo yum install graphviz

    The next step would be creating the control file for graphviz. Therefor we use the normalizer command with the options -d “prefered filename for the control file” and -r “folder of sampledb”

    $ ./normalize -d control.dot -r /home/Test/messages.rb

    Please note that there is no need for an input or output file.
    If you have a look at the control file now you will see that the content is a little bit confusing, but it includes all information, like the nodes, fields and parser, that graphviz needs to create the graph. Of course you can edit that file, but please note that it is a lot of work.

    Now we can create the graph by typing

    $ dot control.dot -Tpng >graph.png

    dot + name of control file + option -T -> file format + output file

    That is just one example for using graphviz, of course you can do many other great things with it. But I think this “simple” graph could be very helpful for the normalizer.

    Please find below a sample for such a graph, but please note that this is not such a pretty one. We will update that graph as soon as we have a adequate one. Such a graph can grow very fast by editing your rulebase.

    graph sample
    Click to enlarge.

  • Creating a rulebase

    A first example for a rulebase you can download at
    http://blog.gerhards.net/2010/11/log-normalization-first-results.html

    I will use an excerpt of that rulebase to show you the most common expressions.

    rule=:%date:date-rfc3164% %host:word% %tag:char-to:\x3a%: no longer listening on %ip:ipv4%#%port:number%'

    That excerpt is a common rule. A rule contains different “parts”/properties, like the message you want to normalize (e.g. Host, IP, Source, Syslogtag…)

    All rules have to start with “rule=:”

    The buildup of a property is as follows

    %field name:field type:additional information%

    field name -> that name can be free selected. It should reflect the content of the field, e.g. src-ip for the source IP. In common sense, the field names should be the same in all samples, if the content of the field means the same.

    field type -> selects the accordant parser

    date-rfc3164: date in format of rfc3164

    ipv4: ip adress

    number: sequence of numbers (example: %port:number%)

    word: everything till the next blank (example: %host:word%)

    char-to: the field will be defined by the sign in the additional information (example: %tag:char-to:\x3a%: (x3a means ":" in the additional information))

    additional information -> dependent on the field type; some field types need additional information

    In our example we have some more information that is used as “simple text”. That parts are exactly like the parts in the messages and are not selected by a property.

    Very important:

    In the field type “char-to” you can use any item that is on your keyboard. In the case shown above, the item “:” has to be escaped with it’s ANSII version. Other characters do not have to be escaped.

  • First steps using liblognorm

    Here you can find the first steps to use the pre-release of liblognorm.

    (Please note that the used operating system was Fedora 13.)

    At the moment there are two ways to install libognorm.
    You can install everything you need from git (below you can find all commands you need) or you can download it as tarball at

    libestr
    libee
    liblognorm

    Please note if you install it with tarballs you have to to the same steps which are mentioned below, apart from

    $ git clone git://git.adiscon.com/git/libestr.git
$ autoreconf -vfi

    Installation
    Open a terminal and switch to the folder where you want to install liblognorm. Below you find the necessary commands

    $ git clone git://git.adiscon.com/git/libestr.git

    switch to the new folder libestr

    $ autoreconf -vfi
$ ./configure --libdir=/usr/lib --includedir=/usr/include
$ make
$ make install

    leave that folder and repeat this step for libee

    $ git clone git://git.adiscon.com/git/libee.git

    switch to the new folder libee

    $ autoreconf -vfi
$ ./configure --libdir=/usr/lib --includedir=/usr/include
$ make
$ make install

    leave that folder and repeat this step again for liblognorm

    $ git clone git://git.adiscon.com/git/liblognorm.git

    switch to the new folder liblognorm

    $ autoreconf -vfi
$ ./configure --libdir=/usr/lib --includedir=/usr/include
$ make
$ make install

    That’s all you have to do.

    For a first test we need two further things, a test log and the rulebase. Both can be downloaded at

    http://blog.gerhards.net/2010/11/log-normalization-first-results.html

    After downloading these examples you can use liblognorm. Go to /liblognorm/src and use the command below:

    $ ./normalize -r /home/Test/messages.sampdb -ojson </home/Test/messages.log >/home/Test/temp

    -r = path to the rulebase

    -o = output format

    Please have look at http://www.liblognorm.com/help/available-options-for-normalizer/ for all available options.