LogAnalyzer Basics

This documentation describes some basic concepts of LogAnalyzer. It is meant as a useful guide for understanding concepts, the application itself as well as the rest of the documentation.

Configuring LogAnalyzer

LogAnalyzer is configured via a master configuration file. If the userDB system (see below) is enabled, most settings can be made via the web-application itself without any further need to change the config file.

Intial configuration can also be done during a setup wizard, which writes an initial configuration file itself. This wizard can also enable the userDB system, so in this case you probably never need to touch the configuration file.

Data Sources

A data source is a set of syslog (and other) data that is gathered. Data sources can be text files or databases. Any text file is supported, as long as it contains purely printable characters and LF is used as a line terminator. This applies to all regular text files so in short you can use whatver is present in text format. Obviously, these are files written by the syslogd, but this also includes any other text file, e.g. written by an application as its log file. Note that at this time, a file data source can contain exactly one file (and NOT multiple). For the database, tabels in either MonitorWare format or the format used by php-syslog-ng is supported. We support php-syslog-ng schema mainly for migration scenarios: it is sub-optimal in that it uses text strings where integer values are sufficient. This results in the need for more database space plus slower response time. If you set up something new, be sure the use MonitorWare schema. If you use rsyslog to create the database, please note that rsyslog uses MonitorWare schema by default, too. So you probably need not to do anything special.

Searches

With LogAnalyzer, you can search any datasource for a variety of properties. By default, text is searched within the msg part of the message. However, quite complex searches can be performed. It is suggested to use the "advanced search" button to build these. Alternatively, you can also review the "LogAnalyzer search syntax" documentation to see how to craft complex searches manually.

Note that searches are done via http get requests. That means you can copy and paste an url (or bookmark and email it) and that URL will contain a complete source. This is actually a great way to send searches to a co-worker or have some automatted process (eg via cron " wget) pull specific data on a periodic schedule.

The search part of LogAnalyzer is used much like any major search engine and hopefully is quite intuitive.

Note that a search is currently limited to a single data source only. If you would like to search across different data sources, you need to do this on the basis of individual data sources. To support you at least a bit in the process, there is a special page (for historical reasons called "the Oracle") which generates canned searches for you, so that you only need to click the individual search links to perform these searches. We know this is not perfect, but we hope this is useful.

Context Links and Menues

At appropriate places, LogAnalyzer generates context links to potentially helpful information. For example, links to domain or IP range owner lookup or troubleshooting information (via the external knowledge base) are generated. At other places, links to the cross-datasource search capability are generated.

Consider this example to understand why it is done and how it may be useful. Let's assume you have two data sources, one with your firewall log and one with the mail log. Now you wonder why a recent spam attack could happen. You review the mail log and find indication of the spammer. Now, you can lookup the IP addresses and domain names used. Probably more interesting, you can invoke the cross-datasource search tool and obtain information on what the firewall log has recorded about the IPs in question. Finally, you may want to check the online knowledge based if there is some information recorded about this or a similar event (for example, if you found a message that puzzels you). You may even ask you peers for help via the knowledge base.

In addition to the links, there are helpful popup menues for most of the properties being displayed. To find out what you can do, please simply click the values (even those that do NOT look like a link) and see what the popup has to offer (and, yes, we will improve doc on these topics... ;)).

Windows Event Log Data

LogAnalyzer contains automatic support for displaying Windows Event Log data in a useful format if that data is generated by either the EventReporter or the MonitorWare Agent forwarding agents. This includes proper detection, and ability to filter, on event-log specific properties (like Windows event id and such).

The UserDB System

The so-called userDB system allows different user accounts to be created and user-specific settings to be made. This is a great aid if multiple people share a single instance of LogAnalyzer.

The userDB system is disabled by default. This is because a database is needed for the userDB system as user profiles are stored inside it. Setting up the database tables requires some additonal work, so we do not expect users to do this by default. Please do not confuse this with database (log) data sources: these are not necessarily needed. You can use the userDB system and still store the log data in text files (which may be desirable from a performance point of view). To enable the userDB system, the LogAnalyzer configuration file needs to be changed. Note that once the userDB system is active, most system settings can be made via the web application.

There are basically two types of users: admins and non-admins. Admins can change anything, non-admins can only change their personal preferences.

The userDB system is not yet a strong security tool, but helps greatly with moderate security needs. User groups can be created and data sources be assigned to a specific user group. So only users of this group can access the data source in question. This is useful if you have a group of people caring about the firewall logs and another group caring about the mail logs. You can then define two different groups and assign the data source accordingly. Then, assign the user's the group they should belong to. The end result is that every user only sees what he or she is expected to see.

Do I need a Database?

Not necessarily. LogAnalyzer only needs databases if you plan to use the userDB system or use database data sources. Without that, no database is needed. A typical scenario, for example, is private review of server-based syslog files. For this use case, no database is required.

Securing LogAnalyzer

Log data is very valuable to an attacker. So it is highly suggested that you secure access to any LogAnalyzer instance, especially if it contains live log data. We suggest to place it on local, non-internet accessible servers, only. In a hosting environment, it may be useful to place it on an internet-accessible server. In this case, access should be protected on the http layer at least. In any case, the use of https is suggested to prevent accidental loss of confidentiality (this is important in the local network, too!).

The userDB system can be used as a tool to tune user's ability to view data sources (users can only view those sources that belong to one of their groups). However, this is currently considered a secondary access control mechanism. An Internet-accessible instance of LogAnalyzer should NOT rely on that as the sole source of protection.

Please note that this section gives just a few rough, common-sense recommendations. Evaluate the risk yourself, check with your policies and do not blame us if you made a mistake ;) In short: use LogAnalyzer at your sole risk, and reduce this risk by thinking about what you do.

You may consider to purchase professional services if you are serious about the risk implications in an (enterprise) environment.

External Tool Integration

As already described, LogAnalyzer accesses external tools, most importantly the MonitorWare Knowledge Base to aid you in your analysis effort. The goal is to provide useful information that helps you get the job done better and quicker.

No private data (except the obious one, eg a domain name for a domain search) is provided to the external entity nor is anything recorded (except for what can be seen from regular web logs). However, you need to decide yourself, as with all external accesses, if this functionality fits into your security policy.

Please note that the external tool is able to call back into your local LogAnalyzer installation if you provide it with the local URL. This can be done via the user profile in the external part. Note that both the local as well as the external part are programmed in such a way that no private data (except for the URL) needs to be kept externally and, most important, local data is never visible to a third-party observer. The notable exception is if you host your local LogAnalyzer on an external server without any further security measure (e.g. https, access restrictions). Even in that case the external part will not be able to access data from the local part, but an attacker may find it easy to get hold of your log data (which is always the case with such a local setup and not specifically related to the local/external integration).

Upgrades

LogAnalyzer is actively being developed. So chances are good you want to upgrade to a later release at some time. In general, it is always a good idea to backup everything before you upgrade. We highly recommend doing so. Other than that, the upgrade should be fairly easy without the installed userDB system. If the userDB system is installed, it may be necessary to upgrade the database schema. This can be done via the admin center. No access to LogAnalyzer is possible unless the database schema has been upgraded.

Again, as a general precaution, you should have a solid backup available before you run the upgrade procedure (and this specifically applies to any database content!).

[manual index] [LogAnalyzer site]

This documentation is part of the Adiscon LogAnalyzer project.
Copyright © 2008-2011 by Adiscon. Released under the GNU GPL version 3 or higher. Adiscon LogAnaylzer commercial licenses are also available.