Money Trace Plugin¶
This plugin allows ATS to participate in a distributed tracing system based upon the Comcast “Money” distributed tracing and monitoring library. The Comcast “Money” library has its roots in Google’s Dapper and Twitters Zipkin systems. A money trace header or session id, is attached to transaction and allows an operator with the appropriate logging systems in place, to determine where errors and/or latency may exit.
Use of the library enables the tracing of a transaction through all systems that participate in handling the request. See the documentation on this open source library at https://github.com/Comcast/money.
How it Works¶
This plugin checks incoming requests for the “X-MoneyTrace” header. If the header is not present no further processing takes place. However if the header is present, the plugin will check to see if the request has been cached. If so, the plugin will add the “X-Moneytrace” header from the incoming request to the cached response returned to the client as required by the money_trace protocol. If the request has not been cached, the plugin will extends the trace context by creating a new “X-MoneyTrace” header for inclusion in the outgoing request to a parent cache or origin server. The extended header includes the ‘trace-id’ from the incoming request, the incoming span-id becomes the outgoing parent-id and the plugin generates a new span id for the outgoing request using the current state machine id.
See the documentation at the link above for a complete description on the “X-MoneyTrace” header and how to use and extend it in a distributed tracing system.
A sample money-trace header:
Here is an example remap.config entry:
map http://vod.foobar.com http://origin.vod.foobar.com @plugin=money_trace.so
The plugin supports the following options:
If no X-MoneyTrace header is found in the client request one will be manufactured using the transaction UUID as trace-id, the transaction state machine id as span-id and parent-id set to ‘0’.
--global-skip-header=[header name](default: null/disable)
The global skip check is performed during the post remap phase in order to allow remap plugins (like header rewrite) to set this skip header.
It is strongly suggested to use a private ATS header (begins with ‘@’) as this value.
Allows the money trace header to be overridden.
In this mode ATS acts transparently and passes the client money trace header through to the parent. It also returns this same header back to the client. This option ignores the –create-if-none setting.
--pregen-header=[header name](default: null/disable)
Normally the money trace header for a transaction is only added to the
transaction server request headers. If this argument is supplied the
header will be generated earlier in the transaction and added to the
client request headers. Use this for debug or for logging the current
transaction’s money trace header. It is suggested to use a private
ATS header (begins with a ‘@’) for this value. A
entry with pregen-header=@MoneyTrace might look like:
This plugin tries to be robust in its parsing. At a minimum the value must start with trace-id= set to a none empty value.
If span-id= is found in the header value that will be used as the parent-id for an upstream request. Otherwise ‘0’ will be its value.
If the incoming money trace header is invalid, it is handled based on the –create-if-none setting. If create-if-none is set a new money trace header will be generated and used. Otherwise the incoming client header value will be passed through.