liblognorm 0.1.0 has been released

Thursday, December 9th, 2010

Liblognorm is a event and log normalization library that is capable of real-time processing. It provides the capability to normalize events to a set of standard formats. It is most efficient when used together with almost unstructured data, as for example found in typical syslog messages. While liblognorm provides a service similar to other projects, it is unique in the way it works:

a) As a library, it’s services can be used by any third-party application with ease. As such, other projects can benefit from liblognorm functionality without the user even knowing.

b) Liblognorm is very fast, and especially much faster than regular-expression based solutions. This is possible because it uses a special data structure which (kind of) combines many regular expressions into a single, and thus faster, big one.

This is the initial public release, targeted at early adopters. We will continue to enhance considerably, but the 0.1.0 version offers decent stability and features and so can actually be used.

You can download it here.

Introducing liblognorm

Wednesday, October 13th, 2010

Liblognorm shall help to make sense out of syslog data, or, actually, any event data that is present in text form.

In short words, one will be able to throw arbitrary log message to liblognorm, one at a time, and for each message it will output well-defined name-value pairs and a set of tags describing the message.

So, for example, if you have traffic logs from three different firewalls, liblognorm will be able to “normalize” the events into generic ones. Among others, it will extract source and destination ip addresses and ports and make them available via well-defined fields. As the end result, a common log analysis application will be able to work on that common set and so this backend will be independent from the actual firewalls feeding it. Even better, once we have a well-understood interim format, it is also easy to convert that into any other vendor specific format, so that you can use that vendor’s analysis tool. (more…)