#include <ts/BufferWriter.h>


BufferWriter is designed to make writing text to a buffer fast, convenient, and safe. It is easier and less error-prone than using a combination of sprintf and memcpy as is done in many places in the code.. A BufferWriter can have a size and will prevent writing past the end, while tracking the theoretical output to enable buffer resizing after the fact. This also lets a BufferWriter instance write into the middle of a larger buffer, making nested output logic easy to build.

The header files are divided in to two variants. BufferWriter.h provides the basic capabilities of buffer output control. BufferWriterFormat.h provides formatted output mechanisms, primarily the implementation and ancillary classes for BufferWriter::print().

BufferWriter is an abstract base class, in the style of std::ostream. There are several subclasses for various use cases. When passing around this is the common type.

FixedBufferWriter writes to an externally provided buffer of a fixed length. The buffer must be provided to the constructor. This will generally be used in a function where the target buffer is external to the function or already exists.

LocalBufferWriter is a templated class whose template argument is the size of an internal buffer. This is useful when the buffer is local to a function and the results will be transferred from the buffer to other storage after the output is assembled. Rather than having code like

char buff[1024];
ts::FixedBufferWriter w(buff, sizeof(buff));

can be more compactly and robustly done as:

ts::LocalBufferWriter<1024> w;

In many cases using LocalBufferWriter this is the only place the size of the buffer needs to be specified, and therefore can simply be a constant without the overhead of defining a size to maintain consistency.


The basic mechanism for writing to a BufferWriter is BufferWriter::write(). This is an overloaded method for a character (char), a buffer (void *, size_t) and a string view (string_view). Because there is a constructor for string_view that takes a const char* as a C string, passing a literal string works as expected.

There are also stream operators in the style of C++ stream I/O. The basic template is

template < typename T > ts::BufferWriter& operator << (ts::BufferWriter& w, T const& t);

Several basic types are overloaded and it is easy to extend to additional types. For instance, to make ts::TextView work with BufferWriter, the code would be

ts::BufferWriter & operator << (ts::BufferWriter & w, TextView const & sv) {
   w.write(sv.data(), sv.size());
   return w;


The data in the buffer can be extracted using BufferWriter::data(). This and BufferWriter::size() return a pointer to the start of the buffer and the amount of data written to the buffer. This is very similar to BufferWriter::view() which returns a string_view which covers the output data. Calling BufferWriter::error() will indicate if more data than space available was written. BufferWriter::extent() returns the amount of data written to the BufferWriter. This can be used in a two pass style with a null / size 0 buffer to determine the buffer size required for the full output.


The BufferWriter::clip() and BufferWriter::extend() methods can be used to reserve space in the buffer. A common use case for this is to guarantee matching delimiters in output if buffer space is exhausted. BufferWriter::clip() can be used to temporarily reduce the buffer size by an amount large enough to hold the terminal delimiter. After writing the contained output, BufferWriter::extend() can be used to restore the capacity and then output the terminal delimiter.


Never call BufferWriter::extend() without previously calling BufferWriter::clip() and always pass the same argument value.

BufferWriter::remaining() returns the amount of buffer space not yet consumed.

BufferWriter::auxBuffer() returns a pointer to the first byte of the buffer not yet used. This is useful to do speculative output, or do bounded output in a manner similar to use BufferWriter::clip() and BufferWriter::extend(). A new BufferWriter instance can be constructed with

ts::FixedBufferWriter subw(w.auxBuffer(), w.remaining());

Output can be written to subw. If successful, then w.fill(subw.size()) will add that output to the main buffer. Depending on the purpose, w.fill(subw.extent()) can be used - this will track the attempted output if sizing is important. Note that space for any terminal markers can be reserved by bumping down the size from BufferWriter::remaining(). Be careful of underrun as the argument is an unsigned type.

If there is an error then subw can be ignored and some suitable error output written to w instead. A common use case is to verify there is sufficient space in the buffer and create a “not enough space” message if not. E.g.

ts::FixedBufferWriter subw(w.auxBuffer(), w.remaining());
if (!subw.error()) w.fill(subw.size());
else w << "Insufficient space"_sv;


For example, error prone code that looks like

char new_via_string[1024]; // 512-bytes for hostname+via string, 512-bytes for the debug info
char * via_string = new_via_string;
char * via_limit  = via_string + sizeof(new_via_string);

// ...

* via_string++ = ' ';
* via_string++ = '[';

// incoming_via can be max MAX_VIA_INDICES+1 long (i.e. around 25 or so)
if (s->txn_conf->insert_request_via_string > 2) { // Highest verbosity
   via_string += nstrcpy(via_string, incoming_via);
} else {
   memcpy(via_string, incoming_via + VIA_CLIENT, VIA_SERVER - VIA_CLIENT);
   via_string += VIA_SERVER - VIA_CLIENT;
*via_string++ = ']';


ts::LocalBufferWriter<1024> w; // 1K internal buffer.

// ...

w << " [";
if (s->txn_conf->insert_request_via_string > 2) { // Highest verbosity
   w << incoming_via;
} else {
   w << ts::string_view{incoming_via + VIA_CLIENT, VIA_SERVER - VIA_CLIENT};
w << ']';

Note that in addition there will be no overrun on the memory buffer in w, in strong contrast to the original code.


class BufferWriter

BufferWriter is the abstract base class which defines the basic client interface. This is intended to be the reference type used when passing concrete instances rather than having to support the distinct types.

BufferWriter &write(void *data, size_t length)

Write to the buffer starting at data for at most length bytes. If there is not enough room to fit all the data, none is written.

BufferWriter &write(string_view str)

Write the string str to the buffer. If there is not enough room to write the string no data is written.

BufferWriter &write(char c)

Write the character c to the buffer. If there is no space in the buffer the character is not written.

fill(size_t n)

Increase the output size by n without changing the buffer contents. This is used in conjuction with BufferWriter::auxBuffer() after writing output to the buffer returned by that method. If this method is not called then such output will not be counted by BufferWriter::size() and will be overwritten by subsequent output.

char *data() const

Return a pointer to start of the buffer.

size_t size() const

Return the number of valid (written) bytes in the buffer.

string_view view() const

Return a string_view that covers the valid data in the buffer.

size_t remaining() const

Return the number of available remaining bytes that could be written to the buffer.

size_t capacity() const

Return the number of bytes in the buffer.

char *auxBuffer() const

Return a pointer to the first byte in the buffer not yet consumed.

BufferWriter &clip(size_t n)

Reduce the available space by n bytes.

BufferWriter &extend(size_t n)

Increase the available space by n bytes. Extreme care must be used with this method as BufferWriter will trust the argument, having no way to verify it. In general this should only be used after calling BufferWriter::clip() and passing the same value. Together these allow the buffer to be temporarily reduced to reserve space for the trailing element of a required pair of output strings, e.g. making sure a closing quote can be written even if part of the string is not.

bool error() const

Return true if the buffer has overflowed from writing, false if not.

size_t extent() const

Return the total number of bytes in all attempted writes to this buffer. This value allows a successful retry in case of overflow, presuming the output data doesn’t change. This works well with the standard “try before you buy” approach of attempting to write output, counting the characters needed, then allocating a sufficiently sized buffer and actually writing.

BufferWriter &print(TextView fmt, ...)

Print the arguments according to the format. See bw-formatting.

class FixedBufferWriter : public BufferWriter

This is a class that implements BufferWriter on a fixed buffer, passed in to the constructor.

FixedBufferWriter(void *buffer, size_t length)

Construct an instance that will write to buffer at most length bytes. If more data is written, all data past the maximum size is discarded.

reduce(size_t n)

Roll back the output to n bytes. This is useful primarily for clearing the buffer by calling reduce(0).

FixedBufferWriter auxWriter(size_t reserve = 0)

Create a new instance of FixedBufferWriter for the remaining output buffer. If reserve is non-zero then if possible the capacity of the returned instance is reduced by reserve bytes, in effect reserving that amount of space at the end. Note the space will not be reserved if reserve is larger than the remaining output space.

template<size_t N>
class LocalBufferWriter : public BufferWriter

This is a convenience class which is a subclass of FixedBufferWriter. It which creates a buffer as a member rather than having an external buffer that is passed to the instance. The buffer is N bytes long.


Construct an instance with a capacity of N.

Formatted Output

BufferWriter supports formatting output in a style similar to Python formatting via BufferWriter::print(). This takes a format string which then controls the use of subsquent arguments in generating out in the buffer. The basic format is divided in to three parts, separated by colons.

Format    ::=  "{" [name] [":" [specifier] [":" extension]] "}"
name      ::=  index | name
extension ::=  <printable character except "{}">*

The name of the argument to use. This can be a number in which case it is the zero based index of the argument to the method call. E.g. {0} means the first argument and {2} is the third argument after the format.

bw.print("{0} {1}", 'a', 'b') => a b

bw.print("{1} {0}", 'a', 'b') => b a

The name can be omitted in which case it is treated as an index in parallel to the position in the format string. Only the position in the format string matters, not what names those other format elements may have used.

bw.print("{0} {2} {}", 'a', 'b', 'c') => a c c

bw.print("{0} {2} {2}", 'a', 'b', 'c') => a c c

Note that an argument can be printed more than once if the name is used more than once.

bw.print("{0} {} {0}", 'a', 'b') => a b a

bw.print("{0} {1} {0}", 'a', 'b') => a b a

Alphanumeric names refer to values in a global table. These will be described in more detail someday.


Basic formatting control.

specifier ::=  [[fill]align][sign]["#"]["0"][[min][.precision][,max][type]]
fill      ::=  <printable character except "{}%:"> | URI-char
URI-char  ::=  "%" hex-digit hex-digit
align     ::=  "<" | ">" | "=" | "^"
sign      ::=  "+" | "-" | " "
min       ::=  integer
precision ::=  integer
max       ::=  integer
type      ::=  "x" | "o" | "b"

The output is placed in a field that is at least min wide and no more than max wide. If the output is less than min then

  • The fill character is used for the extra space required. This can be an explicit character or a URI encoded one (to allow otherwise reserved characters).

  • The output is shifted according to the align.


    Align to the left, fill to the right.


    Align to the right, fill to the left.


    Align in the middle, fill to left and right.


    Numerically align, putting the fill between the output and the sign character.

The output is clipped by max width characters or the end of the buffer. precision is used by floating point values to specify the number of places of precision. The precense of the # character is used for integer values and causes a radix indicator to be used (one of 0xb, 0, 0x).

type is used to indicate type specific formatting. For integers it indicates the output radix. If # is present the radix is prefix is generated with case matching that of the type (e.g. type x causes 0x and type X causes 0X).

b binary
o octal
x hexadecimal
Text (excluding braces) that is passed to the formatting function. This can be used to provide extensions for specific argument types (e.g., IP addresses). The base logic ignores it but passes it on to the formatting function for the corresponding argument type which can then behave different based on the extension.

User Defined Formatting

When an value needs to be formatted an overloaded function for type V is called.

BufferWriter& ts::bwformat(BufferWriter& w, BWFSpec const& spec, V const& v)

This can (and should be) overloaded for user defined types. This makes it easier and cheaper to build one overload on another by tweaking the spec as it passed through. The calling framework will handle basic alignment, the overload does not need to unless the alignment requirements are more detailed (e.g. integer alignment operations).

The output stream operator operator<< is defined to call this function with a default constructed BWFSpec instance.


A planned future extension is a variant of BufferWriter that operates on a MIOBuffer. This would be very useful in many places that work with MIOBuffer instances, most specifically in the body factory logic.