Mututal TLS

In addition to the standard TLS where the server side provides a certificate for authentication, Traffic Server supports mututal TLS (mTLS) where the client also provides a certificate to the origin for verification.

mTLS from User Agent to Traffic Server

In this scenario, Traffic Server is acting as the TLS server. It can request during the TLS handshake that the User Agent provides a certificate. If Traffic Server can verify that the user agent provided certificate is signed by a trusted CA, the TLS handshake will proceed. Otherwise it will fail.

Case 1: Require certificates from all User Agents

In this case, you must set the proxy.config.ssl.client.certification_level setting in records.yaml to 2 to require a client certificate from all user agents. Setting this to 0 means that no client certificate is requested. Setting this to 0 means that a client certificate is requested but the handshake proceeds even if one is not provided. There may be problems with some clients and the 1 setting, so staying with values 0 or 2 may be best.

If the certificate_level is set to 2, you must also set proxy.config.ssl.CA.cert.path and proxy.config.ssl.CA.cert.filename in records.yaml to point to a file that contains the certificates of the CA’s that would have signed the user agent provided certificates that Traffic Server receives.

Case 2: Apply different certificate requirements depending on the domain requested by the User Agent

Often there are scenarios where Traffic Server must require client certificates from some user agents (e.g. trusted parter sites) but not others (e.g. health check requests). In that case, use the sni.yaml file.

Traffic Server uses the Server Name Indication (SNI) from the TLS Client Hello to distinguish between the different cases. This is the FQDN value in the sni.yaml file. To control client certificate requirements use the “verify_client” keyword which can take on the following values: NONE, MODERATE, or STRICT.

In the case were Traffic Server should require certificates from all domains except the health check domain, hc.example.com, you should set proxy.config.ssl.client.certification_level to 2 in records.yaml and have the following in sni.yaml.

sni:
- fqdn: hc.example.com
  verify_client: NONE

Similarly, if you only wanted to require client certificates for super.sensitive.example.com, you would set proxy.config.ssl.client.certification_level to 0 in records.yaml and have the following in sni.yaml

sni:
- fqdn: super.sensitive.example.com
  verify_client: STRICT

You can also use wildcards in the fqdn names (e.g. ‘foo.com’ or ‘mail..foo.com’).

Awkward healthcheck case

Above we showed how you can exempt a health check request from needing to provide a client certificate. That technique requires the health check requester to provide a SNI value. Unfortunately many older clients (including many current hardware loadbalancers), do not set the SNI value in the client hello. From Traffic Server’s perspective the SNI value is the empty string. In that case, the following sni.yaml should work. It will match on all requests do not provide a SNI and turn off the client certificate requirement. This is a very broad rule, since it is very easy for a malicious user to make a request without the SNI set to try to evade the requirements of your sni.yaml policy.

sni:
- fqdn: ''
  verify_client: NONE

Specialize CA Bundle for client cert

You can use the verify_client_ca_certs keyword to specialize the CA bundle name in sni.yaml. For example you expect all client certs to be signed by the roots in client_CA_bundle.pem except for special.example.com where the client certs should be signed by roots in partners_bundle.pem. Then you would set proxy.config.ssl.CA.cert.filename to client_CA_bundle.pem in records.yaml and you would set the following in sni.yaml

sni:
- fqdn: special.example.com
  verify_client_ca_certs: partners_bundle.pem
  verify_client: STRICT

Session resumption across co-hosted SNIs

TLS session resumption never asks the client for its certificate again, so a resumed session carries forward whatever peer certificate the original handshake produced. Traffic Server therefore binds every session it issues to the SNI it was issued under together with that SNI’s verify_client mode and depth, and the trust anchors the client certificate was verified against. A session offered under a different SNI, under the same SNI after its policy changed, or after those trust anchors were replaced, is not resumed. The TLS library declines it and proceeds with a full handshake, in which the current policy is applied as usual. This is what RFC 6066 section 3 requires of session ID resumption; RFC 8446 imposes no equivalent requirement on TLS 1.3 tickets, and Traffic Server applies the binding to them because that is what makes verify_client enforceable across resumption.

Two tenants sharing a listener therefore do not accept each other’s sessions, whatever their verify_client levels, except where the binding cannot be established before the TLS library selects a session. That is the case on a build providing neither ClientHello callback, and on Traffic Server built against OpenSSL when a plugin changes client verification from a TS_EVENT_SSL_CERT hook, because OpenSSL creates and looks up the session before those hooks run.

Which changes retire existing sessions depends on how the trust anchors are configured:

A bundle file, from verify_client_ca_certs’s file or from proxy.config.ssl.CA.cert.filename, contributes the certificates it holds. Replacing it in place retires the sessions it verified. For verify_client_ca_certs this takes effect on the next connection; for proxy.config.ssl.CA.cert.filename it takes effect on the next traffic_ctl config reload, because that bundle is read when certificate contexts are built.
A hashed directory, from verify_client_ca_certs’s dir or from proxy.config.ssl.CA.cert.path, contributes only its configured path. Its certificates are read lazily during verification, and reading them on every handshake to discover which are present would cost more than the guarantee is worth. Retiring a trust anchor from such a directory does not retire the sessions it verified; where that matters, use a bundle file.

Because a session can only be resumed under the policy that verified its client certificate, the certificate does not have to be re-validated on resumption. A client certificate issued by an intermediate CA therefore keeps working across resumption with only the root in the applicable CA bundle, exactly as it does on a full handshake. Should a resumed session nonetheless carry no client certificate where verify_client: STRICT requires one, Traffic Server rejects the handshake and increments proxy.process.ssl.user_agent_resumed_cert_rejected.

Guidance for testing

If you use curl to test your SNI-based Traffic Server configuration, you must make sure the SNI value is really set in the TLS Client Hello message. If you use the Traffic Server name or address in the URL and explicitly set the host field (as shown below) to indicate the real domain, the SNI value will not be set to the host field value. In the example below the SNI value will not be foo.com

curl -H 'host:foo.com' -k -v https://prod123.example.com/foo

You can use the -resolve option of curl to ensure the sni value is set as shown below or update your local /etc/hosts so the address for your designed domain is the proxy address.

curl -resolve foo.com:443:1.2.3.4 -k -v https://foo.com/foo

mTLS from Traffic Server to Origin

In this scenario Traffic Server is the TLS client talking to the upstream origin.

Case 1: Provide one certificate to all potential origins that require a certificate

In this case, you would set at least proxy.config.ssl.client.cert.filename to the name and path of a file that includes the client certificate and the client private key. You could also set proxy.config.ssl.client.cert.path to indicate the path of the file.

The private key could be stored in a separate file named by proxy.config.ssl.client.private_key.filename and proxy.config.ssl.client.private_key.path.

Case 2: Provide different certificates to origins depending on the specific origin name

In this case you would again use the sni.yaml file. The fqdn would correspond to the SNI that Traffic Server passes to the origin. Specifically you would set the client_cert and possibly the client_key values to point to the files containing the client certificate and client keys.

When setting up this case, it is important to understand what value Traffic Server will be using for the SNI name to origin. By default the value of the Host header in the request to origin will be used for the ssl_server_name fqdn lookup. If proxy.config.url_remap.pristine_host_hdr is set, this will be the same host header value as in the user agent request. You can use proxy.config.ssl.client.sni_policy to change Traffic Server to use the remap hostname instead as the fqdn lookup value,

Case 3: Provide different certificates to origins depending on origin name and request URL

In this case you use the conf_remap.so plugin on a remap rule to override the cient_cert definition only for URLs that match that remap rule. You could create the following lines in your remap.config to override the value of proxy.config.ssl.client.cert.filename in records.yaml for specific types of traffic. In the example below any client traffic with a path that starts with /case1 will use the customer-case1.pem certificate. Any client traffic directed to the hostname bank.example.com and a path that starts with /pci will use the pci.pem certificate.

map /case1 https://server.com/case1 @plugin=conf_remap.so @pparam=proxy.config.ssl.client.cert.filename=customer-case1.pem
map /case2 https://server.com/case2 @plugin=conf_remap.so @pparam=proxy.config.ssl.client.cert.filename=customer-case2.pem
map https://bank.example.com/pci https://pci.server.com/ @plugin=conf_remap.so @pparam=proxy.config.ssl.client.cert.filename=pci.pem

Outbound session reuse and client certificates

Two remap rules that present different client certificates can reach the same origin, so a pooled outbound session has to be matched on the certificate as well as the address and hostname. Otherwise a keep-alive session that authenticated with one certificate can serve a transaction configured for another, and the origin continues to authorize it as the first identity because reuse performs no second handshake.

proxy.config.http.server_session_sharing.match covers this by default: its default value of both includes cert. If you set that record explicitly, include cert whenever a remap rule or a plugin overrides proxy.config.ssl.client.cert.filename, for example ip,host,cert. A value that omits cert, such as ip or ip,host, disables the check.

cert tracks only proxy.config.ssl.client.cert.filename and its per-transaction overrides. It does not see a client certificate selected by a sni.yaml client_cert entry, because that path picks the outbound context directly rather than through the overridable record. Those entries are keyed on the outbound SNI, so sni is the value that separates the identities they select; include it when sni.yaml chooses the certificate.

Guidance for testing

You will want to verify that Traffic Server will accurately reload to pick up new client certificate files. As time goes one, the life time of certificates shrink from months to weeks or days, so you will most likely need to have Traffic Server reload configurations to load up new certificates without restarting the Traffic Server process (and interrupting customer traffic). The following command should cause updated client certificates and keys to be loaded into the traffic_server process. From there you can verify via your origins that the updated certificates are being offered.

traffic_ctl config reload

If the contents of the certificate files change but the names of the files do not, you may need to touch ssl_multicert.config (for server certs) and sni.yaml (for client certs).