storage.yaml

The storage.yaml file (by default, located in /usr/local/etc/trafficserver/) lists all the files, directories, and/or hard disk partitions that make up the Traffic Server cache. After you modify the storage.yaml file the new settings will not be effective until Traffic Server is restarted.

Format

The format of the storage.yaml file is a series of lines of the form

cache:               # file level key
  spans:             #
    - name:          # name of the span
      path:          # path to storage
      size:          # size in bytes, required for file system storage, optional for raw device
      hash_seed:     # optional, used to isolate lookup from path changes
  volumes:           # optional
    - id:            # identifier [1-255]
      size:          # optional, size in bytes or percentage
      scheme:        # optional, default to "http"
      ram_cache:     # optional, default to "true"
      avg_obj_size:  # optional, overrides proxy.config.cache.min_average_object_size
      fragment_size: # optional, overrides proxy.config.cache.target_fragment_size
      spans:         # optional
        - use:       # Span identifier
          size:      # size allocated to this volume

spans lists the raw storage used for the cache. volumes organizes the storage into locations for storing cached objects. This is very similar to operating system partitions and file systems.

For spans the keys are

Key	Type	Meaning
name	string	Name of the span.
path	string	File system of the storage. This must be a block device or directory.
size	bytes	Size in bytes. This is optional for devices but required for directories.
hash_seed	string	Hashing for object location uses a seed to randomize the hash. By default this is the path for the span.

For volumes the keys are

Key	Type	Meaning
id	integer	Id of the volume. Range is [1-255]. This id can be referred from `hosting.config`. Each volume must have a distinct id; volume number 0 is reserved for internal use.
size	bytes _or_ percentage	Target size of the entire volume. This can be an absolute number of bytes or a percentage.
scheme	enumeration string	Protocol scheme, defaults to “http”. Preserved for future use.
ram_cache	boolean	Control of ram caching for this volume. Default is `true`. This may be desirable if you are using something like ramdisks, to avoid wasting RAM and CPU time on double caching objects.
ram_cache_size	bytes	Allocate a dedicated RAM cache pool for this volume. The specified amount is automatically subtracted from the global RAM cache pool; the remainder is shared among volumes without a private allocation. Setting to `0` disables the RAM cache for this volume (equivalent to `ram_cache: false`). If the sum of all per-volume allocations exceeds the global limit, Traffic Server will fail to start with a fatal error. If `ram_cache: false` is also set, this value is ignored with a warning. Only takes effect when the global size is a positive value (not `-1` for automatic sizing).
ram_cache_cutoff	bytes	Overrides the global `proxy.config.cache.ram_cache_cutoff` configuration for this volume. Objects larger than this size will not be placed in the RAM cache for this volume. Use a larger cutoff for volumes serving frequently accessed large objects, a smaller cutoff for volumes with many small objects to maximize RAM cache hits, or a very low cutoff to effectively disable RAM caching for large objects on that volume.
avg_obj_size	integer	Overrides the global `proxy.config.cache.min_average_object_size` configuration for this volume. This is useful if you have a volume that is dedicated for say very small objects, and you need a lot of directory entries to store them.
fragment_size	integer	Overrides the global `proxy.config.cache.target_fragment_size` configuration for this volume. This allows for a smaller, or larger, fragment size for a particular volume. This may be useful together with `avg_obj_size` as well, since a larger fragment size could reduce the number of directory entries needed for a large object. Note that this setting has a maximmum value of 4MB.
spans	list	Spans that provide storage for this volume. Defaults to all spans.

For volumes:spans the keys are

Key	Type	Meaning
use	string	Name of the span to use.
size	bytes _or_ percentage	Amount of the span to use. The total across all uses of this specific span must be less than 100% and less than the total size of the span.

Each volume is striped across several disks to achieve parallel I/O. For example: if there are four disks, then a 1-GB volume will have 256 MB on each disk (assuming each disk has enough free space available). If you do not allocate all the disk space in the cache, then the extra disk space is not used. You can use the extra space later to create new volumes without deleting and clearing the existing volumes.

Important

Any change to this file can (and almost always will) invalidate the existing cache in its entirety.

You can use any partition of any size. For best performance:

Use raw disk partitions.
For each disk, make all partitions the same size.
Group similar kinds of storage into different volumes. For example split out SSD’s or RAM drives into their own volume.

Specify pathnames according to your operating system requirements. See the following examples. In the storage.yaml file, a formatted or raw disk must be at least 128 MB.

When using raw disk or partitions, you should make sure the Traffic Server user used by the Traffic Server process has read and write privileges on the raw disk device or partition. One good practice is to make sure the device file is set with ‘g+rw’ and the Traffic Server user is in the group which owns the device file. However, some operating systems have stronger requirements - see the following examples for more information.

As with standard records.yaml integers, human readable prefixes are also supported. They include

K Kilobytes (1024 bytes)

M Megabytes (1024^2 or 1,048,576 bytes)

G Gigabytes (1024^3 or 1,073,741,824 bytes)

T Terabytes (1024^4 or 1,099,511,627,776 bytes)

Storage Allocation

Allocation of span storage to volumes is done in stages. Storage is always allocated in multiples of 128 megabytes, rounded down.

Explicitly sized span storage (cache:volumes:spans:size) is allocated to volumes. It is an error if the total allocated is larger than the span size. * Absolute sizes are allocated first. * Percentages are allocated from remaining space. * Remaining storage from spans that are used without an explicit size is divided evenly among the volumes that use the span.
Span storage is allocated to volumes by the cache:volumes:size values. * Absolute sizes are allocated first. * Percentages are applied to remaining space. * Remaining storage is divided evenly among volumes without an explicit size.

Assignment Table

Each storage element defined in storage.yaml is divided into stripes. The assignment table maps from an object URL to a specific stripe. The table is initialized based on a pseudo-random process which is seeded by hashing a string for each stripe. This string is composed of a base string, an offset (the start of the stripe on the storage element), and the length of the stripe. By default the path for the storage is used as the base string. This ensures that each stripe has a unique string for the assignment hash. This does make the assignment table very sensitive to the path for the storage elements and changing even one can have a cascading effect which will effectively clear most of the cache. This can be a problem when drives fail and a system reboot causes the path names to change.

The name option can be used to create a fixed string that an administrator can use to keep the assignment table consistent by maintaining the mapping from physical device to base string even in the presence of hardware changes and failures.

Backwards Compatibility

In previous versions of Traffic Server it was possible to have “exclusive” spans which were used by only one volume. This is now done by specifying the span in the volume and using a size of “100%”.

Important

In the old volume.config, absolute sizes were specified in megabytes (e.g., size=512 meant 512 MB). In storage.yaml, sizes are specified in bytes with optional human-readable suffixes (e.g., size: 512M or size: 536870912). A bare number without a suffix is interpreted as bytes, not megabytes.

E.g. old configuration like

/dev/disk2 volume=3 # storage.config
volume=3 scheme=http size=512 # volume.config

The corresponding configuration would be

cache:
  spans:
    - name: disk.2
      path: /dev/disk2
  volumes:
    - id: 3
      spans:
        - use: disk.2
          size: 512M

Because volume sizes that are percentages are computed on span storage not already explicitly allocated, this will leave none of “disk.2” for such allocation and therefore “disk.2” will be used only by volume “1”. Note this configuration is more flexible. If it was useful to have two linear volumes, each using exclusively half of the span, this would be

cache:
  spans:
    - name: disk.2
      path: /dev/disk2
  volumes:
    - id: 1
      spans:
        - use: disk.2
          size: 50%
    - id: 2
      spans:
        - use: disk.2
          size: 50%

Important

If a span is explicitly used by any volume its storage will be allocated to only volumes that explicitly use that span.

Examples

The following basic example shows 128 MB of cache storage in the “/big_dir” directory

cache:
  spans:
    - name: store
      path: /big_dir
      size: 134217728

As an alternative, using the human readable prefixes, you can express a 64GB cache file with

cache:
  spans:
    - name: store
      path: /really_big_dir
      size: 64G

By default a volume uses all spans, therefore a volume uses all of span “store” because there are no other volumes. It would be equivalent to using the spans explicitly, e.g.

cache:
  spans:
    - name: store
      path: /big_dir
      size: 128M
  volumes:
    - id: 1
      size: 100%
      spans:
        - use: store

You can use the . symbol for the current directory. Here is an example for 128 MB of cache storage in the current directory

cache:
  spans:
    - name: store
      path: "."
      size: 128M

Note

When using on-filesystem cache disk storage, you can only have one such directory specified. This will be addressed in a future version.

Linux Example

Note

Rather than refer to disk devices like /dev/sda, /dev/sdb, etc., modern Linux supports alternative symlinked names for disk devices in the /dev/disk directory structure. As noted for the Assignment Table the path used for the disk can affect the cache if it changes. This can be ameliorated in some cases by using one of the alternate paths in via /dev/disk. Note that if the by-id or by-path style is used, replacing a failed drive will cause that path to change because the new drive will have a different physical ID or path.

If this is not sufficient then the hash_seed key should be used to create a more permanent assignment table. An example would be

cache:
  spans:
    - name: "span.0"
      path: "/dev/sde"
      hash_seed: "cache.disk.0"
    - name: "span.1"
      path: "/dev/sdg"
      hash_seed: "cache.disk.1"

The following example will use an entire raw disk in the Linux operating system

cache:
  spans:
    - name: a
      path: "/dev/disk/by-id/disk-A-id"
    - name: b
      path: "/dev/disk/by-id/disk-B-id"
  volumes:
    - id: 1
      spans:
        - use: a
          size: 100%
    - id: 2
      spans:
        - use: b
          size: 100%

In order to make sure traffic_server will have access to this disk you can use udev(7) to persistently set the right permissions. The following rules are targeted for an Ubuntu system, and stored in /etc/udev/rules.d/51-cache-disk.rules:

# Assign DiskA and DiskB to the tserver group
# make the assignment final, no later changes allowed to the group!
SUBSYSTEM=="block", KERNEL=="sd[ef]", GROUP:="tserver"

In order to apply these settings, trigger a reload with udevadm(8):

udevadm trigger --subsystem-match=block

FreeBSD Example

Starting with 5.1 FreeBSD dropped support for explicit raw devices. All devices on FreeBSD can be accessed raw now.

The following example will use an entire raw disk in the FreeBSD operating system

cache:
  spans:
    - name: ada.1
      path: "/dev/ada1"
    - name: ada.2
      path: "/dev/ada2"
  volumes:
    - id: 1
      size: 100%

In order to make sure traffic_server will have access to this disk you can use devfs(8) to persistently set the right permissions. The following rules are stored in devfs.conf(5):

# Assign /dev/ada1 and /dev/ada2 to the tserver user
own    ada[12]  tserver:tserver

Advanced

Because relative paths in storage.yaml are relative to the base prefix, when using customized runroot it may be necessary to adjust such paths in storage.yaml or adjust runroot.yaml itself. Despite the name, the cachedir value is not used for this file.

Examples

The following example partitions the cache across 5 volumes to decrease single-lock pressure for a machine with few drives. The last volume being an example of one that might be composed of purely ramdisks so that the ram cache has been disabled.

cache:
  spans:
    - name: disk
      path: "/dev/sdb"
  volumes:
    - id: 1
      size: 20%
    - id: 2
      size: 20%
    - id: 3
      size: 20%
    - id: 4
      size: 20%
      avg_obj_size: 4096
    - id: 5
      size: 20%
      ram_cache: false

This can be simplified by depending on the default allocation which splits unallocated span storage across volumes.

cache:
  spans:
    - name: disk
      path: "/dev/sdb"
  volumes:
    - id: 1
    - id: 2
    - id: 3
    - id: 4
    - id: 5
      ram_cache: false

For a host with a physical disk and two ram disks, where the ram disks should be split between two volumes, with a third volume that uses the physical disk.

This depends on defaults. The spans “ram.1” and “ram.2” are split evenly between volume “1” and volume “2” because no sizes are specified. Span “disk” is not used for volume “1” nor volume “2” because it is not listed in the spans. Volume “3” therefore gets all of span “disk”.

cache:
  spans:
    - name: disk
      path: "/dev/sdb"
    - name: ram.1
      path: "/dev/ram.1"
    - name: ram.2
      path: "/dev/ram.2"
  volumes:
    - id: 1
      spans:
        - use: ram.1
        - use: ram.2
    - id: 2
      spans:
        - use: ram.1
        - use: ram.2
    - id: 3

If one of the ram disk based volumes should be larger, this could be done as follows by making volume “1” roughly twice as large as volume “2”.

cache:
   spans:
     - name: disk
       path: "/dev/sdb"
     - name: ram.1
       path: "/dev/ram.1"
     - name: ram.2
       path: "/dev/ram.2"
   volumes:
     - id: 1
       spans:
         - use: ram.1
           size: 66%
         - use: ram.2
           size: 66%
     - id: 2
       spans:
         - use: ram.1
         - use: ram.2
     - id: 3

Instead, suppose the physical spans (“disk.1” and “disk.2”) should be split across volumes. This can be done by adding volumes with only defaults, as the physical spans will be divided evenly among four volumes (3 - 6), each volume allocated 25% of “disk.1” and 25% of “disk.2”.

OTOH, the ram spans (“ram.1” and “ram.2”) will be divided evenly among volume 1 and 2.

cache:
   spans:
     - name: disk.1
       path: "/dev/sdb"
     - name: disk.2
       path: "/dev/sde"
     - name: ram.1
       path: "/dev/ram.1"
     - name: ram.2
       path: "/dev/ram.2"
   volumes:
     - id: 1
       spans:
         - use: ram.1
         - use: ram.2
     - id: 2
       spans:
         - use: ram.1
         - use: ram.2
     - id: 3
     - id: 4
     - id: 5
     - id: 6

The following example shows advanced RAM cache configuration with dedicated allocations and custom cutoffs. Assuming a global proxy.config.cache.ram_cache.size of 4GB, volume 1 receives a dedicated 2GB RAM cache pool, volume 2 gets 512MB and only caches objects up to 64KB, and volume 3 shares the remaining 1.5GB (4GB - 2GB - 512MB) and caches objects up to 1MB.

cache:
  spans:
    - name: disk
      path: "/dev/sdb"
  volumes:
    - id: 1
      size: 40%
      ram_cache_size: 2G
    - id: 2
      size: 20%
      ram_cache_size: 512M
      ram_cache_cutoff: 64K
    - id: 3
      size: 40%
      ram_cache_cutoff: 1M