RPC

Overview

In order for programs in different address spaces (remote clients and traffic_manager) to communicate with traffic_server, there is a RPC mechanism in mgmt/.

This is a simple serialization style RPC which runs over unix domain sockets. The following sections will detail the runtime structure, serialization mechanisms and how messages are passed from remote clients to traffic_server.

Runtime Structure

traffic_manager opens a unix domain socket to receive commands from remote clients. traffic_manager also has a unix domain socket connection with traffic_server to communicate.

Message Passing

Sequence diagram for a command sent from traffic_ctl to when it is received by a plugin.

Note

Currently a fire and forget model. traffic_manager sends out an asynchronous signal without any acknowledgment. It then proceeds to send a response itself.

Remote RPC vs Local RPC

The RPC API for remote clients, such as traffic_ctl, etc, is different from the RPC API used between traffic_manager and traffic_server.

traffic_manager acts like a bridge for remote clients to interact with traffic_server. Thus, it is currently impossible to do things like have traffic_ctl directly send messages to traffic_server. Classes suffixed with “Remote”, ie. CoreAPIRemote.cc, and classes suffixed with “Local”, ie. NetworkUtilsLocal.cc are for remote and local clients, respectively. The following sections will note which set of RPC’s are relevant.

Serialization Mechanism

class MgmtMarshall

This is the class used to marshall data objects. It provides functions to marshall and unmarshall data. Each data object is associated with a field. Fields are of MgmtMarshallType:

• MGMT_MARSHALL_INT : 4 bytes.

• MGMT_MARSHALL_LONG : 8 bytes.

• MGMT_MARSHALL_STRING : 4 bytes to indicate the string size in bytes, followed by the entire string and NULL terminator.

• MGMT_MARSHALL_DATA : 4 byt es to indicate data size in bytes, followed by the entire data object.

When data is actually sent over a connection it must first be serialized. This is the general serialization mechanism for RPC communication.

Generally, for remote clients sending messages to traffic_server, the message is serialized using remote RPC APIs. The serialized message is sent to traffic_manager and traffic_manager then simply relays the message to traffic_server. traffic_server eventually unserializes the message.

Details for how traffic_manager and traffic_server communicate are documented in the next section.

Marshalling:

ssize_t mgmt_message_marshall(void *buf, size_t remain, const MgmtMarshallType *fields, unsigned count, ...)

Variable argument wrapper for mgmt_message_marshall_v. Allows for different data types to be marshalled together as long as a field is specified for each data object. Arguments should be references to objects to be marshalled.

ssize_t mgmt_message_marshall_v(void *buf, size_t remain, const MgmtMarshallType *fields, unsigned count, va_list ap)

This function goes through all the data objects and serializes them together into a buffer. Based on the field, the number of bytes is determined and if there is enough space, it is written into the buffer.

Unmarshalling:

ssize_t mgmt_message_parse(const void *buf, size_t len, const MgmtMarshallType *fields, unsigned count, ...)

Variable argument wrapper for mgmt_message_parse_v. Reference to data object to store unmarshalled message needed for variable arguments.

ssize_t mgmt_message_parse_v(const void *buf, size_t len, const MgmtMarshallType *fields, unsigned count, va_list ap)

This function parses all the serialized. Based on the field, the number of bytes to be read is determined and copied into a MgmtMarshallAnyPtr.

Local Serialization

A RPC message is sent as a MgmtMessageHdr followed by the serialized data in bytes. Serialization is very simple as the interface for communication between traffic_manager and traffic_server only allows for MgmtMessageHdr, which is a fixed size, and raw data in the form of char* to be sent. A header specifies a msg_id and the data_len. On a read, the header is always first read. Based on the length of the data payload, the correct number of bytes is then read from the socket. On a write, the header is first populated and sent on the socket, followed by the raw data.

class MgmtMessageHdr

int msg_id

ID for the event or signal to be sent.

int data_len

Length in bytes of the message.

RPC API for remote clients

NetworkMessage.cc provides a set of APIs for remote clients to communicate with traffic_manager.

traffic_manager will then send a response with the return value. The return value only indicates if the request was successfully propagated to traffic_server. Thus, there is no way currently for traffic_server to send information back to remote clients.

TSMgmtError send_mgmt_request(const mgmt_message_sender &snd, OpType optype, ...)

TSMgmtError send_mgmt_request(int fd, OpType optype, ...)

Send a request from a remote client to traffic_manager.

TSMgmtError send_mgmt_response(int fd, OpType optype, ...)

Send a response from traffic_manager to remote client.

TSMgmtError send_mgmt_error(int fd, OpType optype, TSMgmtError error)

Send err from traffic_manager to remote client.

TSMgmtError recv_mgmt_request(void *buf, size_t buflen, OpType optype, ...)

Read request from remote client. Used by traffic_manager.

TSMgmtError recv_mgmt_response(void *buf, size_t buflen, OpType optype, ...)

Read response from traffic_manager. Used by remote clients.

Using Remote RPC API Example

This details how a remote client may use the NetworkMessage.cc interface to send messages to traffic_manager. Leveraging send_mgmt_request() with a mgmt_message_sender which specifies how send a message, a remote client can implement its own way to send messages to traffic_manager to allow for things such as preprocessing messages.

class mgmt_message_sender

class mgmtapi_sender : public mgmt_message_sender

TSMgmtError send(void *msg, size_t msglen) const override

Specifies how the message is to be sent. Can do things such as serialization or preprocessing based on parameters.

#define MGMTAPI_SEND_MESSAGE(fd, optype, ...) send_mgmt_request(mgmtapi_sender(fd), (optype), __VA_ARGS__)

Now, using this macro, messages can easily be sent. For example:

TSMgmtError ret;
MgmtMarshallData reply    = {nullptr, 0};

// create and send request
ret = MGMTAPI_SEND_MESSAGE(main_socket_fd, op, &optype);
if (ret != TS_ERR_OKAY) {
  goto done;
}

// get a response
ret = recv_mgmt_message(main_socket_fd, reply);
if (ret != TS_ERR_OKAY) {
  goto done;
}

RPC API for traffic_server and traffic_manager

LocalManager and ProcessManager follow similar workflows. A manager will poll the socket for any messages. If it is able to read a message, it will handle it based on the msg_id from the MgmtMessageHdr and select a callback to run asynchoronously. The async callback will add a response, if any, to an outgoing event queue within the class. A manager will continue to poll and read on the socket as long as there are messages available. Two things can stop a manager from polling.

1. there are no longer any messages on the socket for a timeout time period.

2. an external event, for example, a traffic_ctl command, triggers an eventfd to tell the manager to stop polling. In the case of an external event, the manager will finish reading anything on the socket, but it will not wait for any timeout period. So, if there are three pending events on the socket, all three will be processed and then immediately after, the manager will stop polling.

Once a manager is done polling, it will begin to send out messages from it’s outgoing event queue. It continues to do this until there are no more messages left. Note that as a manager is polling, it is unable to write anything on the socket itself so the number of messages read will always be exhaustive.

class BaseManager

LocalManager

class LocalManager : public BaseManager

This class is used by traffic_manager to communicate with traffic_server

void pollMgmtProcessServer()

This function watches the socket and handles incoming messages from processes. Used in the main event loop of traffic_manager.

void sendMgmtMsgToProcesses(MgmtMessageHdr *mh)

This function is used by traffic_manager to process the messages based on msg_id. It then sends the message to traffic_server over sockets.

ProcessManager

class ProcessManager : public BaseManager

This class is used by traffic_server to communicate with traffic_manager

int pollLMConnection()

This function periodically polls the socket to see if there are any messages from the LocalManager.

void signalManager(int msg_id, const char *data_raw, int data_len)

This function sends messages to the LocalManager using sockets.

Note

1. Both LocalManager::pollMgmtProcessServer() and ProcessManager::pollLMConnection() actually use select, not poll or epoll, underneath.