logging.config

The logging.config file defines all custom log file formats, filters, and processing options. The file itself is a Lua script.

Important

This configuration file replaces the XML based logs_xml.config from past Traffic Server releases. If you are upgrading from a Traffic Server release which used that configuration file, and you have created custom log formats, filters, and destinations, you will need to update those settings to this format.

Log Definitions

Custom logs are configured by the combination of three key elements: a format, an optional filter, and a log destination.

A format defines how log lines will appear (as well as whether the logs using the format will be event logs or summary logs).

A filter defines what events do, and what events don’t, make it into the logs employing the filter.

A log defines where the record of events or summaries ends up.

Formats

Custom logging formats may be provided directly to a log definition, or they may be defined as a reusable variable in your logging.config for ease of reference, particularly when you may have more than one log using the same format. Which approach you use is entirely up to you, though it’s strongly recommended to create an explicit format object if you intend to reuse the same format for multiple log files.

To create a format object, store the result of the format function in a variable. The function takes a table with two attributes: a mandatory string Format which defines the output format string for every event; and an optional number Interval defining the aggregation interval for summary logs.

-- A one-line-per-event format that just prints event timestamps.
myformat = format {
  Format = '%<cqtq>'
}

-- An aggregation/summary format that prints the last event timestamp from
-- the interval along with the total count of events in the same interval.
-- (Doing so every 30 seconds.)
mysummaryformat = format {
  Format = '%<LAST(cqtq)> %<COUNT(*)>',
  Interval = 30
}

You may define as many and as varied a collection of format objects as you desire.

Format Specification

The format specification provided as the required Format entry of the table passed to the format function is a simple string, containing whatever mixture of logging field variables and literal characters meet your needs. Logging fields are discussed in great detail in the Log Fields section.

Flexible enough to not only emulate the logging formats of most other proxy and HTTP servers, but also to provide even finer detail than many of them, the logging fields are very easy to use. Within the format string, logging fields are indicated by enclosing their name within angle brackets (< and >), preceded by a percent symbol (%). For example, returning to the altogether too simple format shown earlier, the following format string:

'%<cqtq>'

Defines a format in which nothing but the value of the logging field cqtq is interpolated for each event’s entry in the log. We could include some literal characters in the log output by updating the format specification as so:

'Event received at %<cqtq>'

Because the string “Event received at ” (including the trailing space) is just a bunch of characters, not enclosed in %<...>, it is repeated verbatim in the logging output.

Multiple logging fields may of course be used:

'%<cqtq> %<chi> %<cqhm> %<cqtx>'

Each logging field is separately enclosed in its own percent-brace set.

There are a small number of logging fields which extend this simple format, primarily those dealing with request and response headers. Instead of defining a separate logging field name for every single possible HTTP header (an impossible task, given that arbitrary vendor/application headers may be present in both requests and responses), there are instead single logging fields for each of the major stages of an event lifecycle that permit access to named headers, such as:

'%<{User-Agent}cqh>'

Which emits to the log the value of the client request’s User-Agent HTTP header. Other stages of the event lifecycle have similar logging fields: pqh (proxy requests), ssh (origin server responses), and psh (proxy responses).

You will find a complete listing of the available fields in Log Fields.

Aggregation Interval

Every format may be given an optional Interval value, specified as the number of seconds over which events destined for a log using the format are aggregated and summarized. Logs which use formats containing an aggregation interval do not behave like regular logs, with a single line for every event. Instead, they emit a single line only every interval-seconds.

These types of logs are described in more detail in Summary Logs.

Formats have no interval by default, and will generate event-based logs unless given one.

Filters

Filters may be used, optionally, to accept or reject logging for matching events, or to scrub the values of individual fields from logging output (while retaining other information; useful for ensuring that sensitive information cannot inadvertently make it into log files).

Filter objects are created by calling one of the following functions:

filter.accept(string)
Creates a filter object which accepts events for logging which match the rule specified in string.
filter.reject(string)
Creates a filter object which rejects events for logging which match the rule specified in string.
filter.wipe(string)
Creates a filter object which clears the values of query parameters listed in string.

For both accept and wipe filters, the string passed defines a rule in the following format:

<field> <operator> <value>

Filter Fields

The log fields have already been discussed in the Formats section above. For a reference to the available log field names, see Log Fields. Unlike with the log format specification, you do not wrap the log field names in any additional markup.

Filter Operators

The operators describe how to perform the matching in the filter rule, and may be any one of the following:

MATCH
True if the values of field and value are identical. Case-sensitive.
CASE_INSENSITIVE_MATCH
True if the values of field and value are identical. Case-insensitive.
CONTAIN
True if the value of field contains value (i.e. value is a substring of the contents of field). Case-sensitive.
CASE_INSENSITIVE_CONTAIN
True if the value of field contains value (i.e. value is a substring of the contents of field). Case-insensitive.

Filter Values

The final component of a filter string specifies the value against which the name field will be compared. For integer matches, all of the operators are effectively equivalent and require the field to be equal to the given integer. For IP addresses, ranges may be specified by separating the first address and the last of the range with a single - dash, as 10.0.0.0-10.255.255.255 which gives the ranges for the 10/8 network. Other network notations are not supported at this time.

Wiping Filters

Filters created with filter.wipe function differently than the accept and reject filters. Instead of a rule, as described above for those filter types, the wiping filter simply lists the query parameter(s) whose values should be scrubbed before any logging occurs. This prevents sensitive information from being logged by fields which include the query string portion of the request URL. It can also be useful to remove things like cache-busting or inconsequentially variable parameters that might otherwise obfuscate the reporting from log analyzers.

Multiple query parameters may be listed, separated by spaces, though only the first occurence of each will be wiped from the query string if any individual parameter appears more than once in the URL.

Logs

Up to this point, we’ve only described what events should be logged and what they should look like in the logging output. Now we define where those logs should be sent. Three options currently exist for the type of logging output, and each is selected by invoking the appropriate function. All three functions take a single Lua table as their argument, with the same set of key/value pairs.

log.ascii(table)
Creates an ASCII logging object.
log.binary(table)
Creates a binaryy logging object.
log.pipe(table)
Creates a logging object that logs to a pipe.

There is no need to capture the return values of these functions. Which type of logging output you choose depends largely on how you intend to process the logs with other tools, and a discussion of the merits of each is covered elsewhere, in Deciding Between ASCII or Binary Output.

The following subsections cover the contents of the table which should be passed when creating your logging object. Only Filename and Format are required.

Name Type Description
Filename string The name of the logfile relative to the default logging directory (set with proxy.config.log.logfile_dir).
Format string or format obj Either a format object created earlier, or a string with a valid format specification.
Header string If present, emitted as the first line of each new log file.
RollingEnabled see below Determines the type of log rolling to use (or whether to disable rolling). Overrides proxy.config.log.rolling_enabled.
RollingIntervalSec number Interval in seconds between log file rolling. Overrides proxy.config.log.rolling_interval_sec.
RollingOffsetHr number Specifies an hour (from 0 to 23) at which log rolling is guaranteed to align. Only has an effect if RollingIntervalSec is set to greater than one hour. Overrides proxy.config.log.rolling_offset_hr.
RollingSizeMb number Size, in megabytes, at which log files are rolled.
Filters array of filters The optional list of filter objects which restrict the individual events logged.
CollationHosts array of strings If present, one or more strings specifying the log collation hosts to which logs should be delivered, each in the form of “<ip>:<port>”. Log Collation for more information.

Enabling log rolling may be done globally in records.config, or on a per-log basis by passing appropriate values for the RollingEnabled key. The latter method may also be used to effect different rolling settings for individual logs. The numeric values that may be passed are the same as used by proxy.config.log.rolling_enabled. For convenience and readability, the following predefined variables may also be used in logging.config:

log.roll.none
Disable log rolling.
log.roll.time
Roll at a certain time frequency, specified by RollingIntervalSec and RollingOffsetHr.
log.roll.size
Roll when the size exceeds RollingSizeMb.
log.roll.both
Roll when either the specified rolling time is reached or the specified file size is reached.
log.roll.any
Roll the log file when the specified rolling time is reached if the size of the file equals or exceeds the specified size.

Examples

The following is an example of a format that collects information using three common fields:

minimalfmt = format {
  Format = '%<chi> : %<cqu> : %<pssc>'
}

The following is an example of a format that uses aggregate operators to produce a summary log:

summaryfmt = format {
  Format = '%<LAST(cqts)> : %<COUNT(*)> : %<SUM(psql)>',
  Interval = 10
}

The following is an example of a filter that will cause only REFRESH_HIT events to be logged:

refreshhitfilter = filter.accept('pssc MATCH REFRESH_HIT')

The following is an example of a filter that will cause the value of the first query parameter named passwd to be wiped.

passwdfilter = filter.wipe('passwd')

The following is an example of a log specification that creates a local log file for the minimal format defined earlier. The log filename will be minimal.log because we select the ASCII logging format.

log.ascii {
  Filename = 'minimal',
  Format = minimalfmt
}

The following is an example of a log specification that creates a local log file using the summary format from earlier, and only includes events that matched the REFRESH_HIT filter we created.

log.ascii {
  Filename = 'refreshhit_summary',
  Format = summaryfmt,
  Filters = { refreshhitfilter }
}

Further Reading

As the logging.config configuration file is just a Lua script (with a handful of predefined functions and variables), general Lua references may be handy for those not already familiar with the language.