Converting records.config to records.yaml

Since ATS 10 records.config is not longer used and instead we have migrated to records.yaml. This is a guide on how to migrate from your legacy records.config style file to a new records.yaml.

This document will give details about the change and how to convert files from the legacy style to the new YAML.

New YAML structure

A YAML node records` is the main root node for the records.yaml, so all existing records will belong to this records` node.

From the current records.config structure we have dropped the prefix proxy.config and proxy.local and use what's left on the variable name to build a YAML structure, always below the records node.

The logic around this is basically to walk down the record name separated by the dots and build a new YAML map from each name, so for instance, with the following records:

CONFIG proxy.config.exec_thread.autoconfig.scale FLOAT 1.0
CONFIG proxy.config.exec_thread.limit INT 2
CONFIG proxy.config.accept_threads INT 1
CONFIG proxy.config.task_threads INT 2
CONFIG proxy.config.cache.threads_per_disk INT 8
CONFIG proxy.config.exec_thread.affinity INT 1
CONFIG proxy.config.diags.debug.enabled INT 0
CONFIG proxy.config.diags.debug.tags STRING http|dns

we first drop the proxy.config. and walk down each field, so we will end up with the following:

records:
   accept_threads: 1
   cache:
      threads_per_disk: 8
   diags:
      debug:
         enabled: 0
         tags: http|dns
   exec_thread:
      affinity: 1
      autoconfig:
         scale: 1.0
      limit: 2
   task_threads: 2

There were a few things that were done on the names and types that are described next.

Records renaming

There are few changes that I had to perform in order to be able to keep the similar structure used in the records.config into a YAML based configuration.

An example of this would be a record key holding a value which also contains children fields:

proxy.config.exec_thread.autoconfig INT 1
proxy.config.exec_thread.autoconfig.scale FLOAT 1.0 << children of autoconfig

As this cannot be done in YAML we need to rename proxy.config.exec_thread.autoconfig to proxy.config.exec_thread.autoconfig.enabled so we no longer have a map with values other than some children's fields. So in YAML we can go from something like this:

proxy.config.exec_thread.autoconfig INT 1
proxy.config.exec_thread.autoconfig.scale FLOAT 1.0

to

exec_thread:
   autoconfig:
      enabled: 1 # new field.
      scale: 1.0

Naming convention

All this renamed records are subject to the following reasoning:

If a value is meant to be used as a toggle on/off, We have used the name enabled If a value is meant to be used a an enumeration, We have used the name value

Possible issue with this is that if we scale the field meaning and we need to move from a toggle to an enumeration then this logic will make no sense. Should we just use one or the other? Current diags.debug.enabled is an example of an enable field holding more than 0,1 value.

Converting files

We have provided a tool script that will help converting any records.config style file into a YAML format file. The script will not only convert record by record but it will also work around the names that were renamed.

Just run the script on your existing records.config file and that should be it.

Example 1

Converting a file with a detailed output.

$ python3 convert2yaml.py -f records.config -o records.yaml
[████████████████████████████████████████] 494/494

┌■ 8 Renamed records:
└┬──» #1 : proxy.config.output.logfile -> proxy.config.output.logfile.name
 ├──» #2 : proxy.config.exec_thread.autoconfig -> proxy.config.exec_thread.autoconfig.enabled
 ├──» #3 : proxy.config.hostdb -> proxy.config.hostdb.enabled
 ├──» #4 : proxy.config.tunnel.prewarm -> proxy.config.tunnel.prewarm.enabled
 ├──» #5 : proxy.config.ssl.TLSv1_3 -> proxy.config.ssl.TLSv1_3.enabled
 ├──» #6 : proxy.config.ssl.client.TLSv1_3 -> proxy.config.ssl.client.TLSv1_3.enabled
 ├──» #7 : proxy.config.ssl.origin_session_cache -> proxy.config.ssl.origin_session_cache.enabled
 └──» #8 : proxy.config.ssl.session_cache -> proxy.config.ssl.session_cache.value

There are a few things to note here:

Line 2. A total of 494 from 494 records were converted. Line 4. A total of 8 records were renamed.

Example 2

Converting a file with no output. If any, errors are displayed.

$ convert2yaml.py -f records.config -o records.yaml -m

注釈

Use -m, --mute to mute the output.

Non core records

With non core records the problem arises when a plugin register a new field which is not registered in the core, the legacy implementation would read the record type from the config file and then once the record is registered by a plugin it will be properly registered inside ATS( * mismatched type gets the current value reset to the default one*)

As said before in ATS when using YAML without explicit saying the type we have no way to know the type till the registration happens(after reading the config). We need to overcome this and to me there seems to be two options here:

Expect non core records to set the type (!!int, !!float, etc).

records:
   plugin_x:
      my_field_1: !!int '1'
      my_field_2: !!float '1.2'
      my_field_3: 'my string'

The convert2yaml tool can deal with this:

Example 1

Use a type representation (-t, --typerepr) when converting a file.

Non core records expect the type to be specified as the YAML library used Internally cannot assume the type. So in for cases like this you should set the type on the non core records, for instance:

CONFIG proxy.config.http.my_own_record_1 FLOAT 1.0
CONFIG proxy.config.http.some_core_record INT 1

$ convert2yaml.py -f records.config -o records.yaml -t float,int

$ cat records.yaml
records:
   http:
      my_own_record_1: !!float '1.0'
      my_own_record_2: !!int '1'

Now the records parser knows the type of the non core records.

重要

The issue with using type representer when converting a file is that this applies to all the records and not only the non core one. This will change in a future script update.

Final Notes

Internally ATS still uses variables with record style names, querying any record can still be done as usual. Either by using the ATS API or by using traffic_ctl.

ATS only accepts a records.yaml config file, if a legacy file records.config is found then ATS will fail to start.

traffic_server WARNING: **** Found a legacy config file (/your/ats/records.config). Please remove it and migrate to the new YAML format before continuing. ****