Network Connection Timeouts

Every network connection can have an inactivity timeout (fire if nothing has happened for N seconds) and an active timeout (fire N seconds after the connection opened, regardless of activity). InactivityCop is the continuation that dispatches both. One instance runs on each ET_NET thread, scheduled every proxy.config.net.inactivity_check_frequency seconds (default 1).

Deadlines live on the NetEvent as absolute timestamps, next_inactivity_timeout_at and next_activity_timeout_at, where 0 means "no timeout". NetHandler::_earliest_deadline reduces the pair to the single deadline the machinery below acts on: the non-zero minimum, or 0 if both are 0.

The timer wheel

InactivityCop finds expired connections through a TimerWheel owned by the thread's NetHandler (include/tscore/TimerWheel.h). The wheel is a ring of intrusive lists — buckets — one per second. Registering a connection puts it in the bucket for its deadline; each tick drains only the buckets that have come due.

The cost of a tick is therefore proportional to the number of connections whose deadline arrived, not to the number of connections open on the thread. That is the whole point: the previous implementation rebuilt a work list by walking every open connection once per second, which at 100,000 connections per thread cost on the order of 10 ms per tick doing nothing useful.

The wheel holds N_BUCKETS - 1 ticks of range — about 68 minutes at the default 4096 buckets. A deadline beyond that is clamped into the furthest reachable bucket and re-inserted when that bucket comes around, so a long timeout costs one extra visit per ring traversal rather than being wrong. The bucket count is a template parameter; 4096 was chosen by measurement, because at every candidate size the common 30-second and 120-second timeouts cost exactly one visit per timeout period, and only multi-hour timeouts (tunnel active timeouts) care.

Lazy rearm, and the one rule that matters

The wheel caches a copy of each element's deadline. That cached deadline may be earlier than the element's true deadline, but it must never be later, or the timeout fires late.

This asymmetry is what makes the design cheap:

Extending a deadline requires no wheel work at all. The cached value is merely early; when the bucket comes due the wheel re-reads the real deadline, sees it is still in the future, and relinks the element instead of firing it. This is lazy rearm, and it is why UnixNetVConnection::netActivity — which runs on every read and write — does no wheel bookkeeping in the common case.

Setting, shortening, or closing must re-arm, via NetEvent::rearm_timer. Rather than reason about this site by site, every place that writes a deadline calls rearm_timer(), with one deliberate exception noted below. If you add a new write site, call it.

Two consequences that are easy to get wrong

A closed connection must be reaped on the next tick, not at its deadline. NetHandler::rearm_timer maps ne->closed to a deadline of now. The old implementation swept every connection each second, so it noticed a closed one within a tick; the wheel only visits what is scheduled, so without this a closed connection whose deadline is minutes away would hold its file descriptor until then.

A connection with no deadline is not in the wheel at all. If _earliest_deadline() is 0, rearm_timer() cancels rather than schedules. So a connection that has no deadline yet cannot be "noticed later" by the cop the way the old sweep would have noticed it — nothing will visit it until something schedules it. This is the trap behind the default-timeout handling below.

The default inactivity timeout

proxy.config.net.default_inactivity_timeout is a catch-all applied to connections that have no explicit inactivity timeout of their own. It is stored per-connection in default_inactivity_timeout_in, seeded from the global in NetHandler::startCop and overridable per transaction.

The deadline it implies is armed by NetEvent::arm_default_inactivity_timeout, which applies the old sweep's rule: an enabled, open connection with no inactivity deadline gets the default, replacing any explicit timeout. It is called wherever a connection can be left in that state:

  • UnixNetVConnection::set_enabled, when a VIO is first enabled and no explicit timeout is set;

  • UnixNetVConnection::set_inactivity_timeout, when the explicit timeout is cleared with 0;

  • NetEvent::rebase_default_inactivity_deadline, when the default changes on a connection that had none armed (e.g. a per-remap default replacing a global 0);

  • the end of UnixNetVConnection::mainEvent, when the handler kept the connection after a delivered timeout cleared its deadline;

  • InactivityCop, for a connection with no inactivity deadline that an active timeout brought into the cop.

UnixNetVConnection::cancel_inactivity_timeout also leaves the default active by design, and netActivity re-arms from the default instead of clearing the deadline when there is no explicit timeout.

This matters because the only thing that used to apply the default was the cop's own sweep, gated on a VIO being enabled. Under the wheel that does not work: a connection with no deadline is never scheduled, so the cop never visits it, so the default is never applied — the connection would never time out at all.

netActivity() is also the one place that deliberately skips rearm_timer(), since extending is free — except on the 0-to-armed transition. The arming points above should leave no enabled connection without a deadline, but one that reaches netActivity unarmed may not be scheduled, and extending a deadline it does not have would leave it out of the wheel forever, so that single transition still re-arms. Everything after it is a plain extension and stays free.

Budget

A tick's worth of due connections is normally small, but a correlated wave — an origin outage, a load balancer draining — can make it large, and the callbacks run inline on the poll thread. InactivityCop::TIMEOUT_BUDGET caps how many timeouts one pass will fire.

Firing is not the only work, though. A connection whose deadline was extended after it was scheduled is examined and re-armed without firing, so a bucket of connections that were all scheduled in the same second and have all stayed busy would never trip the timeout budget. A pass therefore also stops after examining VISIT_BUDGET_FACTOR (16) times its timeout budget. A re-arm is far cheaper than a close, hence the generous multiple. Hitting either limit counts as hitting the budget.

The budget bounds how long a single pass can hold the thread; it is not a cap on how much work a tick may retire. A pass that hits the budget reschedules itself a millisecond out rather than waiting out the rest of the tick, so a large burst is spread across several trips through the event loop instead of being delayed by whole seconds of inactivity_check_frequency.

The short delay is load-bearing, and schedule_imm() would not do. An immediate event is dispatched inline by EThread::process_queue()'s own dequeue loop, so rescheduling immediately re-enters the pass before the poll events run and bounds nothing. A timed event goes into EventQueue, where dequeue_ready() will not return it during the current loop iteration, so the tail handler gets to poll first.

That distinction matters more than it first appears, because expire() does not advance its cursor until a bucket drains completely. Waiting for the next periodic check would delay not just the remainder of the over-full bucket but every tick queued behind it, so a single large burst could push timeout lag far past the configured interval.

Observability

Testing

The wheel itself is a dependency-free container with unit tests in src/tscore/unit_tests/test_TimerWheel.cc — it needs no event system, so its edge cases (lazy rearm, cancel, wraparound past the ring's range, budget deferral, reentrant rescheduling from a fire callback) are tested directly.

InactivityCop has a microbenchmark at src/iocore/net/unit_tests/benchmark_InactivityCop.cc that drives the real cop against mock NetEvent objects at up to 100,000 connections. It is tagged [!benchmark] so ctest skips it; run it with:

./test_net '[inactivity_cop]'

The mock NetEvent takes no locks and never re-arms on its own, so the paths that live in UnixNetVConnection (the VIO lock-miss retry in mainEvent, the default arming rules, and netActivity's 0-to-armed re-arm) are tested against a real UnixNetVConnection in src/iocore/net/unit_tests/test_UnixNetVConnectionTimeouts.cc (./test_net '[unixnetvc]').

End-to-end behavior is covered by the autests in tests/gold_tests/timeout/. Those are the real gate for anything in this document — in particular default_inactivity_timeout.test.py for the arming rules above. They need a Linux environment that permits socket binds.