liblognorm 2.0.4 released

Wednesday, October 4th, 2017

We have just released liblognorm 2.0.4. This new version mainly provides new parser support options, like different JSON number formats and unix timestamps. See the Changelog for details.

Version 2.0.4, 2017-10-04

  • added support for native JSON number formats supported by parsers: number, float, hex
  • added support for creating unix timestamps supported by parsers: date-rfc3164, date-rfc5424
  • fixed build problems on Solaris
    … but there still seem to be some code issues, manifested in testbench failures. So use with care!

Download:
http://www.liblognorm.com/download/liblognorm-2-0-4/

As always, feedback is appreciated.

Best regards,
Florian Riedl

liblognorm 2.0.3 released

Thursday, March 23rd, 2017

We have just released liblognorm 2.0.3. This new version provides some fixes for the the annotate function and adds a test for it. A few different issues have also been fixed. See the Changelog for details.

Changelog:Version 2.0.3, 2017-03-22

  • add ability to load rulebase from a string
    introduces new API:
    int ln_loadSamplesFromString(ln_ctx ctx, const char *string);
    closes https://github.com/rsyslog/liblognorm/issues/239
  • bugfix: string parser did not correctly parse word at end of line
  • bugfix: literal parser does not always store value if name is specified
    if
    rule=:%{“type”:”literal”, “text”:”a”, “name”:”var”}%
    is used and matching message is provided, variable var ist not persisted.
    see also http://lists.adiscon.net/pipermail/rsyslog/2016-December/043985.html

Download:
http://www.liblognorm.com/download/liblognorm-2-0-3/

As always, feedback is appreciated.

Best regards,
Florian Riedl

Rulebase

Friday, December 3rd, 2010

A rulesbase is a collection of rules. The rulebase is the core of the normalizing process as it holds the information that is needed to transform logs into a common format.

Upon execution of the normalizer, it will be transferred into a parse-tree.

Rule

Friday, December 3rd, 2010

A rule is a a scheme that fits to a specific type of logfile from a specific device. It consists of multiple fields, which reflect the type of information. Many of these rules together build the rulebase.

Creating a rulebase

Tuesday, November 16th, 2010

A first example for a rulebase you can download at
http://blog.gerhards.net/2010/11/log-normalization-first-results.html

I will use an excerpt of that rulebase to show you the most common expressions.

rule=:%date:date-rfc3164% %host:word% %tag:char-to:\x3a%: no longer listening on %ip:ipv4%#%port:number%'

That excerpt is a common rule. A rule contains different “parts”/properties, like the message you want to normalize (e.g. Host, IP, Source, Syslogtag…)

All rules have to start with “rule=:

The buildup of a property is as follows

%field name:field type:additional information%

field name -> that name can be free selected. It should reflect the content of the field, e.g. src-ip for the source IP. In common sense, the field names should be the same in all samples, if the content of the field means the same.

field type -> selects the accordant parser

date-rfc3164: date in format of rfc3164

ipv4: ip adress

number: sequence of numbers (example: %port:number%)

word: everything till the next blank (example: %host:word%)

char-to: the field will be defined by the sign in the additional information (example: %tag:char-to:\x3a%: (x3a means ":" in the additional information))

additional information -> dependent on the field type; some field types need additional information

In our example we have some more information that is used as “simple text”. That parts are exactly like the parts in the messages and are not selected by a property.

Very important:

In the field type “char-to” you can use any item that is on your keyboard. In the case shown above, the item “:” has to be escaped with it’s ANSII version. Other characters do not have to be escaped.