Using text log files with LogAnalyzer

LogAnalyzer natively supports operting on text log files. So there is no need to put log data into databases. Each approach has advantages and disadvantages, which we will not discuss in this paper. Here, we focus on how text log files are supported.

We assume that the reader is familiar with basic LogAnalyzer concepts. If you are not, get aquainted to them, first.

Requirements for text log files

LogAnalyzer can operate on any standard text log file, that means a file that purely consists of printable characters and uses LF (linefeed, newline) characters as log line terminators. Most importantly, this includes standard syslog files as well as web server and other application logs.

A data source is defined to pull data from a text log file. Note that only a single text log file can be contained in any data source.

Working with text log files

Text log files are just regular data sources. You can work with them like with any other data sources, for example, you can browse them, search them, generate graphics from the or base reports on their content. Depending on the size of the text log and on the operation being performed, performance may be worse or even better than with using database-based data sources. For typical operations and typical text file sizes, performance should be quite well.

Note that some options make limited sense when working with text log files. For example, you may set a filter with a broad date range, but the text log file will, for obvious reasons, only provide what is currently contained in it. For typical syslog files, that will probably mean they contain maximum one day's data.

Use Cases for Text Log Files

A typical use case is in a hosting environment. There, LogAnalyzer may be used to do an online review of log data, via the web and without the need to log into an SSH session. This can be very handy to learn what is currently going on or, for example, to verify if an important mail left your system.

Another use case is that you would like to keep an eye on some local text files, for example while debugging an installation. Instead of reconfiguring your whole logging system, you can set up a LogAnalyzer instance and point it to the log files in question. Keep in mind that you are not restricted to syslog files. For example, you may want to add additional application text log files as data sources as well. With the cross-datasource search capability, you can quickly follow events across the boundaries of log servers.

For secuity sensitive-environments, the access to LogAnalyzer can be protected by several levels, e.g. user authentication or IP ranges at the firewall or http server level. Also, one may consider to activate a LogAnalyzer instance only when actually needed to do troubleshooting and keep it "turned off" at other times. The beauty of text log files is that you do not need to reconfigure the whole logging system to do so. All that is needed is to disable the web site inside the http server.

Support of Dynamic Logfilenames (and Pathes)

Since Loganalyzer Version 2.7.0, dynamic logfilenames are supported. You can add date replacement variables into the filename property which will automatically be applied using the current systemtime. For example if you have logfiles located in subfolders named by Year, month and date like this: /var/log/servername/2015/01/30/syslog, the configured filename property would be this: /var/log/Servers/ServerName/%Y/%m/%d/syslog. Below is a list of possible replacements.

%y = Year with two digits (e.g. 2002 becomes "02")
%Y = Year with 4 digits
%m = Month with two digits (e.g. March becomes "03")
%M = Minute with two digits
%d = Day of month with two digits (e.g. March, 1st becomes "01")
%h = Hour as two digits
%S = Seconds as two digits. It is hardly believed that this ever be used in reality.
%w = Weekday as one digit. 0 means Sunday, 1 Monday and so on.
%W = Weekday as three-character string. Possible values are "Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat".

[manual index] [LogAnalyzer site]

This documentation is part of the Adiscon LogAnalyzer project.
Copyright © 2008-2011 by Adiscon. Released under the GNU GPL version 3 or higher. Adiscon LogAnaylzer commercial licenses are also available.