Category: Help

Log Normalization and special characters
When trying to normalize log messages via liblognorm and mmnormalize, you need to create a rulebase first. The rulebase is usually a representation of message formats.
Due to the format of these rules, it is necessary to be cautious. Some messages and rule necessities could possibly cause confusion to the configuration interpreter. This mainly applies to clear text passages in single rules.
For example, if you have a log message from a Cisco ASA, the message looks like this:
```
2012-11-23T10:47:42+01:00 10.10.10.10 : %ASA-3-313001: ...
```
The only interesting parts are the IP and the numerical code to identify the message. We are not interested in the timestamp or “%ASA”. But when making the rule, the trouble starts there. The percent character is also used to define variables and their values in a rule. Thus it needs to be escaped. This is done with the ASCII code representation of the percent character. The rule would look like this:
```
rule=: %date:word% %host:ipv4% : x25ASA-%char1:char-to:-%-%char2:number%: ...
```
If you write “%ASA” into the rule, the interpreter will think, that a new variable starts there. This will cause confusion to the rest of the rule and render it not working correctly. This needs to be avoided.
The same applies to “:”. But this time, it needs to be escaped when using it as delimiter vor variables. Example:
```
%variable:char-to:x3a%
```
This will fill “variable” with everything until the next “:” occurs. If you just put a “:” here as a delimiter, the rule will not work anymore.
November 28, 2012
Log normalization and the leading space
When using log normalization, nothing is simple. But we have been asked something about a very common scenario. A log message has been sent to rsyslog. The message itself had no irregular characters. But, the message that should have been parsed by mmnormalize now has a leading space character. Basically, the message that should be parsed looks like this:
```
This is a test
```
Usually, one would think, that a simple parser can be used here. You might be correct, but there is a small caveat about this. The rulebase entry we currently have looks something like this:
```
rule=:%word1:word% %word2:word% %word3:word% %word4:word%
```
But strangely, rsyslog responds the following:
```
mmnormalize generated: {“originalmsg”: ” This is a test″, “unparsed-data”: ” This is a test″}
```
How comes, that rsyslog cannot parse the message? Why is there a leading space character in from of the message? The answer is, that messages are processed as RFC3164. In this RFC it is defined, that everything after the “:” of the syslog header is to be considered as the message. Thus, the message has a leading space now.
How is this to be solved? Simply insert the space to your rules in the rulebase. This will lead to a rule like this:
```
rule=: %word1:word% %word2:word% %word3:word% %word4:word%
```
Please note, that there has just the space character been added. Further, this is really only a example. The rule will fit to all messages that are 4 words long, so it is really not very suitable to be adopted to your configuration.
June 6, 2011
Using rsyslog mmnormalize module effectively with Adiscon LogAnalyzer
Using the mmnormalize module in rsyslog is a bit complicated at first. We want to describe in this article how to set up the basic components for using log normalization. In addition to that we will show how to configure these components so messages will be split into pieces of information. These pieces of information should then be written into a database for review with Adiscon LogAnalyzer.
This guide has been tested with rsyslog v5.8.0 and liblognorm 0.3, libee 0.3.
The goal of this guide is to have a setup, that will have a message parsed by the normalizing tool, put some content of the message into specific properties. These properties will then be filled into a special database format, which will should be reviewed by Adiscon LogAnalyzer.
For using normalization we need the following:
- rsyslog
- liblognorm
- libee
- libestr
In the further process of the article we need additional elements:
- apache webserver with PHP5
- mysql database (usually with phpmyadmin)
- Adiscon LogAnalyzer
Step 1: Setting up rsyslog and log normalization
First of all we need to setup rsyslog for log normalization. So before installing rsyslog, we will install liblognorm, libee and libestr. They can be installed according to this guide. rsyslog can now be installed. We assume you have downloaded and extracted a tarball from the rsyslog download page. Change into the directory you installed rsyslog in. Now use the following commands to setup rsyslog correctly:
```
./configure --libdir=/lib --sbindir=/sbin --enable-mysql --enable-mmnormalize
make
make install
```
If everything is correct, the installation procedure should complete successfully. We can now start configuring rsyslog itself. We need a configuration that looks like this:
```
$ModLoad immark
$ModLoad imuxsock
$ModLoad imklog
$ModLoad mmnormalize
$ModLoad ommysql.so
$ModLoad imudp.so
```
```
$UDPServerRun 514
```
```
$mmnormalizeUseRawMSG 1
$mmnormalizeRuleBase /rsyslog/rulebase.rb
*.* :mmnormalize:
```
```
$template database,"insert into normalized (date, uhost, msgnumber, protocol, ipin, ipout, portin, portout)
values ('%$!date%', '%$!uhost%', '%$!msgnumber%', '%$!protocol%', '%$!ipin%', '%$!ipout%', '%$!portin%',
'%$!portout%')",SQL
```
```
*.* :ommysql:172.19.3.17,syslog, test, test;database
```
That is all for our rsyslog config. Looks pretty complicated right now. Basically, we load all necessary modules at the top. After that we start the UDP syslog server. It is needed to receive the messages. The next 3 lines are the parameters to initiate the normalization of messages. We declare, that the raw message should be used. Our rulebase for the normalization lies in the rsyslog directory (this path has to be changed if your directory lies somewhere else). And after that, we tell rsyslog to use normalization on all messages. The next line describes the template for the processed message. In the end, there should be a sql insert statement that puts all the parsed variables into their corresponding fields in the table “normalized”. The last line is finally the action that makes rsyslog write all messages (the ones created by the template – the sql statement – into a remote database.
After the configuration, we still need to setup a rulebase. This is done in a separate file. For our example, the rulebase should be the following file: /rsyslog/rulebase.rb
The file should look like this:
```
rule=:%date:date-rfc3164% %uhost:word% %tag:word%  %notused:chat-to:x3a%: %msgnumber:char-to:x3a%: access-list
inside_access_in permitted %protocol:word% inside/%ipin:ipv4%(portin:number%) ->
outside/%ipout:ipv4%(portout:number%) %notused2:char-to:]%]
```
The rule is basically one line. It might be shown otherwise here due to restrictions of the webdesign. It is basically a format of a message. The different parameters of a rule are shown in a different guide. The rule we have here should resemble the following message:
```
May 16 07:23:09 BHG-FW : %ASA-4-106100: access-list inside_access_in  permitted tcp inside/10.200.22.183(2969) ->
outside/67.192.232.82(80)  hit-cnt 1 first hit [0x48e9c345, 0x386bad81]
```
If you want to have multiple messages, where the format differs, you need multiple rules as well of course. The rules must be as precise as possible to resemble the message. If a message does not fit any listed rules, it will not be processed further. Something else that needs to be pointed out, is to keep the rules variable enough as well. Like in our example, there are some parts that will be the same for every message of this kind. Other parts might be with different content. And even if we do not need the content further, it should be put into a variable. Else the message might again not fit to the rule.
Step 2: Setting up the database
We suppose, that you already have a server with a database and webserver installed. The installation of the components has to be made according to the instructions given by the manufacturer of the software. Therefore we cannot give any examples for that.
But we need a specific database scheme for our example here. So we need to show this at least. As you have seen before, we have some specific parts of the message filled into properties. These properties should be written to the database. So here is the basic SQL statement to create the table according to our needs:
```
CREATE TABLE normalize
(
ID int unsigned not null auto_increment primary key,
date datetime NULL,
host varchar(255) NULL,
msgnumber varchar(20),
protocol varchar(60) NULL,
ipin varchar(60) NULL,
ipout varchar(60) NULL,
portin int NULL,
portout int NULL
)
```
You can execute this statement as you like. It is currently designed for a MySQL database, so you might need to change some bits if you are using a different database.
3. Using Adiscon LogAnalyzer with this database
Adiscon LogAnalyzer can be used to review the data from this database. Installation of Adiscon LogAnalyzer is shown here. Please note, that we will need the admin center. So please think of creating a user database when installing.
Point your browser to your Adiscon LogAnalyzer installation. Now we need to go to the admin center. There we have to set some parts to fit our custom format.
Edit Fields
First, we need to add some Fields. We need to do this, so we can use the custom fields in our database with LogAnalyzer. By default, the list of fields only reflects basically the MonitorWare Database Scheme. When clicking on Fields in the Admin Center, a list of the currently available fields will be shown:
By clicking on Add new Field, we can create a new Field.
We need to create 7 new fields only, though we have 8 custom fields in the table. Since date is the same, we can use the already formatted field. So we only need to create the fields for host, msgnumber, protocol, ipin, ipout, portin and portout.
Basically, the Field details should look like this:
To finally create the Field, click on the button “Add new Field”. Now the list should appear again with the newly created Field. Repeat this step for the other fieldnames as well.
Edit DBMappings
In conjunction with the Fields which are only for the internal use in Adiscon LogAnalyzer, we need to create a custom database mapping. Therefore go to DBMappings in the Admin Center. You will see a list of the currently available database mappings.

Click on Add new Database Mapping:

Here we need to tell Adiscon LogAnalyzer, which Field we created depends on which database field. Give your database mapping a name first. After that, choose the Fields we need from the dropdown menu and click on “Add Field Mapping into list”. The final step will be to enter the database field names into the list. It should now look like this now:

Finally click on “Add new Database Mapping”. This will save the mapping and get you back to the list of DBMappings.

Edit Views
The next step we need to adjust is the Views. In Views you can configure, what LogAnalyzer should show. This is related to the data that is stored in the database. Basically, a View should represent the kind of logs that are stored. For example if you use the View for Windows Event Logs, but have a database where Linux syslog is stored, many Fields will be shown as empty, because they are not filled like from Windows Event Logs. Therefore we need a custom view.
You will get there by clicking on Views in the Admin Center.

There are already pre-configured Views for Windows EventLog, Syslog and Webserver Logs. We need a completely different View though. A new View can be configured by clicking on “Add new View” at the bottom of the list.

You need to give your view a name. If you want, you can restrict the use of this view to certain users or groups, but we will skip that for now. The most important part is to select the Fields that should be displayed. This is done at “Configured Columns”. Before clicking on “Add new View” it should look like this:

After clicking the button, the new View should appear in our list.

Edit Sources
Finally, we need to create a Source. When installing Adiscon LogAnalyzer, you can already configure a Source. For our example, we need to create another Source. Therefore go to Sources in the Admin Center.

You will see a list of the configured Sources. It currently holds one Source. By clicking on Add new Source you can create another one.

Basically, we need to insert a Source Name. If you want, you can also create a description. Change the Source Type to MYSQL Native. You can also select a default View. Choose our lognorm View we created earlier. No more general options need to be set. If you want, you can again restrict the source to a user or group.
We still need to change the database Type options. As you can see, the fields have changed by setting the Source Type to MYSQL Native. As table type choose the lognorm type we created before. Insert the details as your database needs them. The complete form should look like this now:

Finish the new Source by clicking on Add new Source. It should now appear in the list.

Final Thoughts
Though this scenario seems very complex it shows in the end how easy some things can be afterwards. This setup shows exactly, how different products from the Adiscon product line can work together. And we have a good example for how normalizing works.
May 20, 2011
Available options for normalizer
-r = path to the rulebase
-o = output format (Encoder) (just in V 0.1.0)
-e = output format (Encoder) (since V 0.2.0) !!!
-E = here you insert the fields that should be dispended (-E “host tag” -> that only dispend the host and the tag field) by default all fields will be dispended
-p = just the parsed messages will be dispensed (since V 0.2.0)
-v = debug outout (-v is the normal debug mode; -vv is an expanded debug mode with more information)
-d = dot file (Is used for creating a graph of the rulebase)
December 17, 2010
Creating a graph of the rulebase
To get a better overview of a rulebase you can create a graph that shows you the chain of normalization.
At first you have to install an additional package called graphviz. Graphviz is a tool that creates such a graph with the help of a control file (created with the rulebase). Here you will find more informaton about graphviz.
To install it you can use the package manager or the yum command.
$ sudo yum install graphviz
The next step would be creating the control file for graphviz. Therefor we use the normalizer command with the options -d “prefered filename for the control file” and -r “folder of sampledb”
$ ./normalize -d control.dot -r /home/Test/messages.rb
Please note that there is no need for an input or output file.
If you have a look at the control file now you will see that the content is a little bit confusing, but it includes all information, like the nodes, fields and parser, that graphviz needs to create the graph. Of course you can edit that file, but please note that it is a lot of work.
Now we can create the graph by typing
$ dot control.dot -Tpng >graph.png
dot + name of control file + option -T -> file format + output file
That is just one example for using graphviz, of course you can do many other great things with it. But I think this “simple” graph could be very helpful for the normalizer.
Please find below a sample for such a graph, but please note that this is not such a pretty one. We will update that graph as soon as we have a adequate one. Such a graph can grow very fast by editing your rulebase.

Click to enlarge.
December 2, 2010
Creating a rulebase
A first example for a rulebase you can download at
http://blog.gerhards.net/2010/11/log-normalization-first-results.html
I will use an excerpt of that rulebase to show you the most common expressions.
rule=:%date:date-rfc3164% %host:word% %tag:char-to:\x3a%: no longer listening on %ip:ipv4%#%port:number%'
That excerpt is a common rule. A rule contains different “parts”/properties, like the message you want to normalize (e.g. Host, IP, Source, Syslogtag…)
All rules have to start with “rule=:”
The buildup of a property is as follows
%field name:field type:additional information%
field name -> that name can be free selected. It should reflect the content of the field, e.g. src-ip for the source IP. In common sense, the field names should be the same in all samples, if the content of the field means the same.
field type -> selects the accordant parser
date-rfc3164: date in format of rfc3164
ipv4: ip adress
number: sequence of numbers (example: %port:number%)
word: everything till the next blank (example: %host:word%)
char-to: the field will be defined by the sign in the additional information (example: %tag:char-to:\x3a%: (x3a means ":" in the additional information))
additional information -> dependent on the field type; some field types need additional information
In our example we have some more information that is used as “simple text”. That parts are exactly like the parts in the messages and are not selected by a property.
Very important:
In the field type “char-to” you can use any item that is on your keyboard. In the case shown above, the item “:” has to be escaped with it’s ANSII version. Other characters do not have to be escaped.
November 16, 2010
First steps using liblognorm
Here you can find the first steps to use the pre-release of liblognorm.
(Please note that the used operating system was Fedora 13.)
At the moment there are two ways to install libognorm.
You can install everything you need from git (below you can find all commands you need) or you can download it as tarball at
libestr
libee
liblognorm
Please note if you install it with tarballs you have to to the same steps which are mentioned below, apart from
```
$ git clone git://git.adiscon.com/git/libestr.git
$ autoreconf -vfi
```
Installation
Open a terminal and switch to the folder where you want to install liblognorm. Below you find the necessary commands
```
$ git clone git://git.adiscon.com/git/libestr.git
```
switch to the new folder libestr
```
$ autoreconf -vfi
$ ./configure --libdir=/usr/lib --includedir=/usr/include
$ make
$ make install
```
leave that folder and repeat this step for libee
```
$ git clone git://git.adiscon.com/git/libee.git
```
switch to the new folder libee
```
$ autoreconf -vfi
$ ./configure --libdir=/usr/lib --includedir=/usr/include
$ make
$ make install
```
leave that folder and repeat this step again for liblognorm
```
$ git clone git://git.adiscon.com/git/liblognorm.git
```
switch to the new folder liblognorm
```
$ autoreconf -vfi
$ ./configure --libdir=/usr/lib --includedir=/usr/include
$ make
$ make install
```
That’s all you have to do.
For a first test we need two further things, a test log and the rulebase. Both can be downloaded at
http://blog.gerhards.net/2010/11/log-normalization-first-results.html
After downloading these examples you can use liblognorm. Go to /liblognorm/src and use the command below:
```
$ ./normalize -r /home/Test/messages.sampdb -ojson </home/Test/messages.log >/home/Test/temp
```
-r = path to the rulebase
-o = output format
Please have look at http://www.liblognorm.com/help/available-options-for-normalizer/ for all available options.
November 15, 2010