Caudium 1.0, 1.2, 1.3

UltraLog is a powerful Logging program that allows you to keep a log history without having to save huge log files. It has some advantages over most of the other logging programs in that you can quickly display daily detail of the logs, or summarize them for a week or month.

Setting up Ultralog

Setting up Caudium/Apache to write logs in an appropriate format

UltraLog can work with log files created by any Web Server, though it works best when working with log files of a finite length, that is, log files that have been "rolled over" and are no longer being written to. Additionally, there are number of fields that allow UltraLog to extract additional data for summarization. This section describes setting up log files for both Caudium and Apache.

With Caudium, if you want to get the best results from logging, you will need to go into the Configuration Interface, into the Virtual Server that you want to use UltraLog on, Go into Server Variables, Click on the Logging... option.

For your Log file Name, you will want to change it so that it has an extension of %y%m%d. This will tell Caudium to open a new log file each day.

i.e. /var/log/caudium-1.1/ratethefunny.com/Log.%y%m%d

You will also want to change the Format of the log file in the Configuration Interface to:

*: $host - $user [$cern_date] "$method $resource $protocol" $response $length "$referer" "$agent_unquoted"

I typically remove the other two log entries for the 404: and 500: specific logging as the Combined logging format above puts the response code into each log entry.

For Apache, you can use the combined format:

CustomLog /var/log/apache/ratethefunny.com-access.log combined

Setting up an UltraLog Config file for your domain

First, you need to create an ultralog config file. UltraLog uses a configuration file to control the summarization and presentation of log data. Typically, the file is called ultralog.cfg, and may be located anywhere convenient. The config file depends greatly on the way that you have logging set up on Caudium or Apache. The format of the ultralog.cfg file is fairly straightforward. There are some field definitions that are used in the Log Format statement.

A sample ultralog.cfg file follows:

<savedir path="/var/colocation/ultralog/clientname/">
<table maxsize="10000">

<profile name="ratethefunny.com">
<file
path="/var/log/caudium/ratethefunny.com*"
format='%H %j %U [%D/%M/%Y:%h:%m:%s %j] "%j %f %j" %c %b "%R" "%U"'
restore>
<extensions list="html htm rxml shtml cgi pike php">
</profile>

The format of the ultralog.cfg file is as follows:

savedir path: This is where the resulting compiled log results are stored. This directory will need to have permissions set so that the webserver can write to the files.

table maxsize: This is basically used as a runaway limit on the size of the data file that is kept.

profile name: This is what will be displayed in the browser when someone views their UltraLog Statistics. Note that you can have multiple profiles within one ultralog.cfg file.

file path: This is the path to the log files that Caudium is generating.

file format: This is the macro substitution language that is used so that UltraLog can read multiple log file formats. The format statement above allows you to read an apache Combined Log file or the suggested Caudium Log entry above.

The macro fields are specified as:

	  %D	Day
	  %H	Host
	  %M	Month
	  %R	Referrer
	  %U	UserAgent
	  %Y	Year
	  %b	Bytes transferred
	  %c	Response Code (200, 302, 404, etc)
	  %f	File Requested
	  %h	Hour
	  %j	Junk - basically 'filler' to let ultrasum know to not use this field
	  %m	Minute
	  %s	Second
	  %u	Username - I am not sure that user logging is enabled by default

	  

extensions list: This is the list of filename extensions that serve pages to the surfer. Typically you want to include any extension that is not a graphic image displayed to a surfer.

You can have multiple <profile> blocks within one ultralog.cfg file and UltraSum will process them one at a time.

Setting up the UltraLog server module

In the Configuration Interface of Caudium, Select the virtual server configuration that you want to run UltraLog under. Go to Add Modules and add UltraLog: Main Module. There are two pieces of information that are fairly important:

Mount Point: This controls the url that you will see when you want to view the stats. You need to make sure this doesn't overlap some other filesystem in your caudium configuration. It defaults to /ultra/ which would make the URL to access your stats: http://domain.com/ultra/

Profile File Name: This is the complete path and filename to the ultralog.cfg file that you created earlier.

After you have changed those, Save the Changes. In order for UltraParse.pike to read those changes, you need to reload the module. Click the reload button and hope that the config interface comes back. Sometimes the Config Interface will report that the Requested Action could not be taken. This usually means that there is a mistake in the ultrasum.cfg file. Check the ultralog.cfg file closely to see if you have a missing " or missing > or haven't closed the <profile> tag with a </profile>.

At this point, you should be able to go to http://domain.com/ultra/ where domain.com is your web site domain and /ultra/ corresponds with the Mount Point you have chosen. If all is well, you should see the site(s) profile names present on the page.

Running UltraSum

UltraSum is the part of UltraLog that digests WebServer log files into a database format that UltraLog uses to display log data on the fly to a client. UltraSum should be run periodically (daily, weekly, etc) to update log data.

UltraSum is run using the following syntax:

ultrasum /path/to/ultralog.cfg

The ultrasum program is installed in the bin/ directory of the prefix where Caudium was installed. For instance, if the Caudium install prefix is /usr/local (the Caudium server directory is /usr/local/caudium/server), ultrasum would be located in /usr/local/bin.

This will cause UltraSum to use the ultralog.cfg file and process the logs based on the directives in that file. UltraSum runs through the log files very quickly and will create directories equivalent to the profile names that you have specified in the ultralog.cfg. You need to make sure that these are chmod'ed so that the web server has write access to them. You might set up a privileged log processing account that shares the same group as the web server and set the sticky bit on the directories to ensure that files created are writeable by the Caudium Webserver. If you run your Caudium Webserver as root, which you shouldn't do for security reasons, you won't need to worry about the permission issues.

If you are using Caudium, and your log files have the extension of %y%m%d, or are guaranteed to have unique names, you can run ultrasum multiple times per day from cron.

If you are running UltraSum on Apache log files, you will need to make sure that ultrasum is not run on a log file more than once. The situation here is that apache doesn't provide an easy way to log to filenames that correspond to a date, so access_log becomes access_log.0 after the rotation, and access_log becomes the new log file. If ultrasum is run on access_log, it does not know that it may have already processed the data and will give you errant results. In short, if you are going to run ultrasum on apache log files, run them on the .0 file after your logs have been rotated, and run ultrasum ONLY once per day.

At this point, logs should run properly and ultrasum should update its database and http://domain.com/ultra/ should have updated statistics depending how frequently you run ultrasum.

Potential Issues

If Caudium does not have write permissions to the directory specified in the path statement of the ultralog.cfg file, it will cause Caudium to execute a traceback with a message like:

Error Message: Error: The server failed to fulfill your query, due to an internal error.

The solution here is to make sure the webserver has write permissions to the directory.