Mutexes

A mutex is the basic synchronization method used within Traffic Server to protect data from simultaneous access by multiple threads. A mutex acts as a lock that protects data in one program thread from being accessed by another thread.

The Traffic Server API provides two functions that attempt to access and lock the data: TSMutexLockTry() and TSMutexLock(). TSMutexLock is a blocking call - if you use it, you can slow Traffic Server performance because transaction processing pauses until the mutex is unlocked. It should be used only on threads created by the plugin TSContThreadCreate. Never use it on a continuation handler called back by the Cache, Net, or Event Processor. Even if the critical section is very small, do not use it. If you need to update a flag, then set a variable and/or use atomic operations. If TSMutexLock() is used in any case other than the one recommended above, then the result will be a serious performance impact.

TSMutexLockTry, on the other hand, attempts to lock the mutex only if it is unlocked (i.e., not being used by another thread). It should be used in all cases other than the above mentioned TSMutexLock case. If the TSMutexLockTry attempt fails, then you can schedule a future attempt (which must be at least 10 milliseconds later).

In general, you should use TSMutexLockTry instead of TSMutexLock.

• TSMutexLockTry is required if you are tying to lock Traffic Server internal or system resources (such as the network, cache, event processor, HTTP state machines, and IO buffers).

• TSMutexLockTry is required if you are making any blocking calls (such as network, cache, or file IO calls).

• TSMutexLock might not be necessary if you are not making blocking calls and if you are only accessing local resources.

The Traffic Server API uses the TSMutex type for a mutex. There are two typical uses of mutex. One use is for locking global data or data shared by various continuations. The other typical usage is for locking data associated with a continuation (i.e., data that might be accessed by other continuations).

Locking Global Data

The denylist_1.cc sample plugin implements a mutex that locks global data. The denylist plugin reads sites to be denied from a configuration file; file read operations are protected by a mutex created in TSPluginInit(). The denylist_1.cc code uses TSMutexLockTry() instead of TSMutexLock(). For more detailed information, see the denylist_1.cc code; start by looking at the TSPluginInit() function.

General guidelines for locking shared data are as follows:

1. Create a mutex for the shared data with TSMutexCreate().

2. Whenever you need to read or modify this data, first lock it by calling TSMutexLockTry(); then read or modify the data.

3. When you are done with the data, unlock it with TSMutexUnlock(). If you are unlocking data accessed during the processing of an HTTP transaction, then you must unlock it before calling TSHttpTxnReenable().

Protecting a Continuation’s Data

You must create a mutex to protect a continuation’s data if it might be accessed by other continuations or processes. Here’s how:

1. Create a mutex for the continuation using TSMutexCreate.

   For example:

   TSMutex mutexp;
   mutexp = TSMutexCreate ();
   

2. When you create the continuation, specify this mutex as the continuation’s mutex.

   For example:

   TSCont contp;
   contp = TSContCreate (handler, mutexp);
   

If any other functions want to access contp’s data, then it is up to them to get contp’s mutex (using, for example, TSContMutexGet) to lock it. For usage, see the sample Protocol plugin.

How to Associate a Continuation With Every HTTP Transaction

There could be several reasons why you’d want to create a continuation for each HTTP transaction that calls back your plugin.

Some potential scenarios are listed below.

• You want to register hooks locally with the new continuation instead of registering them globally to the continuation plugin.

• You want to store data specific to each HTTP transaction that you might need to reuse across various hooks.

• You’re using APIs (like TSHostLookup) that will call back the continuation with a certain event.

How to Add the New Continuation

A typical way of adding the new continuation is by registering the plugin continuation to be called back by HTTP transactions globally when they reach TS_HTTP_TXN_START_HOOK. Refer to the example below, which uses a transaction-specific continuation called txn_contp.

void TSPluginInit(int argc, const char *argv[])
{
    /* Plugin continuation */
    TSCont contp;
    if ((contp = TSContCreate (plugin_cont_handler, nullptr)) == TS_ERROR_PTR) {
        LOG_ERROR("TSContCreate");
    } else {
        if (TSHttpHookAdd (TS_HTTP_TXN_START_HOOK, contp) == TS_ERROR) {
            LOG_ERROR("TSHttpHookAdd");
        }
    }
}

In the plugin continuation handler, create the new continuation txn_contp and then register it to be called back at TS_HTTP_TXN_CLOSE_HOOK:

static int plugin_cont_handler(TSCont contp, TSEvent event, void *edata)
{
    TSHttpTxn txnp = static_cast<TSHttpTxn>(edata);
    TSCont txn_contp;

    switch (event) {
        case TS_EVENT_HTTP_TXN_START:
            /* Create the HTTP txn continuation */
            txn_contp = TSContCreate(txn_cont_handler, nullptr);

            /* Register txn_contp to be called back when txnp reaches TS_HTTP_TXN_CLOSE_HOOK */
            if (TSHttpTxnHookAdd (txnp, TS_HTTP_TXN_CLOSE_HOOK, txn_contp) == TS_ERROR) {
                LOG_ERROR("TSHttpTxnHookAdd");
            }

            break;

        default:
            TSAssert(!"Unexpected Event");
            break;
    }

    if (TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE) == TS_ERROR) {
        LOG_ERROR("TSHttpTxnReenable");
    }

    return 0;
}

Remember that the txn_contp handler must destroy itself when the HTTP transaction is closed. If you forget to do this, then your plugin will have a memory leak.

static int txn_cont_handler(TSCont txn_contp, TSEvent event, void *edata)
{
    TSHttpTxn txnp;
    switch (event) {
        case TS_EVENT_HTTP_TXN_CLOSE:
            txnp = static_cast<TSHttpTxn>(edata);
            TSContDestroy(txn_contp);
            break;

        default:
            TSAssert(!"Unexpected Event");
            break;
    }

    if (TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE) == TS_ERROR) {
        LOG_ERROR("TSHttpTxnReenable");
    }

    return 0;
}

How to Store Data Specific to Each HTTP Transaction

For the example above, store the data in the txn_contp data structure - this means that you’ll create your own data structure. Now suppose you want to store the state of the HTTP transaction:

struct ContData {
      int state;
  };

You need to allocate the memory and initialize this structure for each HTTP txnp. You can do that in the plugin continuation handler when it is called back with TS_EVENT_HTTP_TXN_START

static int plugin_cont_handler(TSCont contp, TSEvent event, void *edata)
{
    TSHttpTxn txnp = static_cast<TSHttpTxn>(edata);
    TSCont txn_contp;
    ContData *contData;

    switch (event) {
        case TS_EVENT_HTTP_TXN_START:
            /* Create the HTTP txn continuation */
            txn_contp = TSContCreate(txn_cont_handler, nullptr);

            /* Allocate and initialize the txn_contp data */
            contData = (ContData*) TSmalloc(sizeof(ContData));
            contData->state = 0;
            if (TSContDataSet(txn_contp, contData) == TS_ERROR) {
                LOG_ERROR("TSContDataSet");
            }

            /* Register txn_contp to be called back when txnp reaches TS_HTTP_TXN_CLOSE_HOOK */
            if (TSHttpTxnHookAdd (txnp, TS_HTTP_TXN_CLOSE_HOOK, txn_contp) == TS_ERROR) {
                LOG_ERROR("TSHttpTxnHookAdd");
            }

            break;

        default:
            TSAssert(!"Unexpected Event");
            break;
    }

    if (TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE) == TS_ERROR) {
        LOG_ERROR("TSHttpTxnReenable");
    }

    return 0;
}

For accessing this data from anywhere, use TSContDataGet:

TSCont txn_contp;
ContData *contData;

contData = TSContDataGet(txn_contp);
if (contData == TS_ERROR_PTR) {
    LOG_ERROR("TSContDataGet");
}
contData->state = 1;

Remember to free this memory before destroying the continuation:

static int txn_cont_handler(TSCont txn_contp, TSEvent event, void *edata)
{
    TSHttpTxn txnp;
    ContData *contData;
    switch (event) {
        case TS_EVENT_HTTP_TXN_CLOSE:
            txnp = static_cast<TSHttpTxn>(edata);
            contData = TSContDataGet(txn_contp);
            if (contData == TS_ERROR_PTR) {
                LOG_ERROR("TSContDataGet");
            } else {
                TSfree(contData);
            }
            TSContDestroy(txn_contp);
            break;

        default:
            TSAssert(!"Unexpected Event");
            break;
    }

    if (TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE) == TS_ERROR) {
        LOG_ERROR("TSHttpTxnReenable");
    }

    return 0;
}

Using Locks

You do not need to use locks when a continuation has registered itself to be called back by HTTP hooks and it only uses the HTTP APIs. In the example above, the continuation txn_contp has registered itself to be called back at HTTP hooks and it only uses the HTTP APIs. In this case only, it’s safe to access data shared between txnp and txn_contp without grabbing a lock. In the example above, txn_contp is created with a nullptr mutex. This works because the HTTP transaction txnp is the only one that will call back txn_contp, and you are guaranteed that txn_contp will be called back only one hook at a time. After processing is finished, txn_contp will re-enable txnp.

In all other cases, you should create a mutex with the continuation. In general, a lock is needed when you’re using iocore APIs or any other API where txn_contp is scheduled to be called back by a processor (such as the cache processor, the DNS processor, etc.). This ensures that txn_contp is called back sequentially and not simultaneously. In other words, you need to ensure that txn_contp will not be called back by both txnp and the cache processor at the same time, since this will result in a situation wherein you’re executing two pieces of code in conflict.

Special Case: Continuations Created for HTTP Transactions

If your plugin creates a new continuation for each HTTP transaction, then you probably don’t need to create a new mutex for it because each HTTP transaction (TSHttpTxn object) already has its own mutex.

In the example below, it’s not necessary to specify a mutex for the continuation created in txn_handler:

 static void
 txn_handler (TSHttpTxn txnp, TSCont contp) {
     TSCont newCont;
     ....
         newCont = TSContCreate (newCont_handler, nullptr);
     // It's not necessary to create a new mutex for newCont.

     ...

         TSHttpTxnReenable (txnp, TS_EVENT_HTTP_CONTINUE);
 }

static int
test_plugin (TSCont contp, TSEvent event, void *edata) {
    TSHttpTxn txnp = static_cast<TSHttpTxn>(edata);

    switch (event) {
        case TS_EVENT_HTTP_READ_REQUEST_HDR:
            txn_handler (txnp, contp);
            return 0;
        default:
            break;
    }
    return 0;
}

The mutex functions are listed below:

• TSMutexCreate()

• TSMutexLock()

• TSMutexLockTry()