State Management Audit for Live Execution System

Situation

Risq operated a live execution system with high unit test coverage and a recurring pattern of production incidents. Individual incidents were small in impact but frequent, roughly one every two weeks, and each required manual intervention. The pattern was consistent: the test suite passed cleanly, the system was redeployed, and a similar-shaped incident recurred within the following sessions.

Internal diagnosis had pointed at concurrency, external data feed reliability, and specific code paths. Each had been investigated and partially addressed without materially changing the incident rate. The rate resisted every remediation because every remediation targeted a different category of failure than the one actually producing the incidents.

State Coverage Problem

The operational problem was not insufficient code coverage. It was that the system was being tested against fresh state while capital was being run against accumulated state.

Diagnosis

The decisive diagnostic was a state-carry test. We snapshotted end-of-day production state from a non-production replica, loaded it into a staging environment, and replayed recorded market data from the following morning against the system. The system failed inside a single morning session.

The failure mode was not a concurrency bug or a data feed anomaly. The failure mode was that the system’s behavior under accumulated state differed from its behavior under fresh state, and the test suite exclusively exercised fresh state. Code-path coverage was high. State-space coverage was not a measured dimension at all.

Three categories of state accumulation were implicated:

• partial fills carried overnight
• position reconciliation across restarts
• order state surviving network partitions

Partial fills were correct intra-session and divergent across session boundaries. Reconciliation behaved correctly on a warm system and incorrectly against persisted pre-restart state. Local order state failed to converge correctly with venue state after reconnect under specific timing, and the suite never simulated the disconnect path.

Intervention

Unit test coverage was not increased. Coverage was already adequate for the logic being tested. A new test category was introduced alongside the existing suite.

A state-persistent test harness was implemented. Its primitives were:

• a session object with explicit begin, restart, end, and overnight transitions
• a state snapshot mechanism that captured and restored the full system state across transitions

Tests were then written against this harness for the session-boundary and restart scenarios the existing suite had never produced.

Test inputs were changed from synthetic to replayed. Recorded market data from representative days was stored and replayed against the harness. Synthetic inputs were retained for logic verification but removed from the state-sensitive path, because synthetic inputs do not reliably produce the pathological input sequences that cause state-accumulation failures.

Continuous invariant checking was added in production. A named catalogue of invariants, including conservation of quantity, consistency between subsystems’ representations of the same position, and reconciliation agreement, was asserted on every state mutation. An invariant breach froze the affected subsystem and paged on-call. Freezing was chosen over auto-recovery because an invariant breach means the system’s model of reality is wrong, and wrong systems should not continue trading.

An invariant catalogue document was produced and maintained. Each invariant had a name, description, implication of breach, and defined operator response. The catalogue was the artifact, not an afterthought.

Outcome

• Incident rate over the six months following remediation fell to roughly one-fifth of the prior baseline.
• Remaining incidents skewed toward genuinely novel failure modes rather than recurrences.
• Mean time to diagnose incidents dropped substantially because the invariant catalogue pre-classified the failure type.
• The state-persistent harness became a permanent part of the development process, required alongside unit tests before merge.

Applicable Pattern

Unit test coverage and integration coverage do not measure state coverage. A system with excellent line coverage can still have a largely unexercised state space.

The diagnostic signal is blunt: if the test suite cannot survive a restart mid-session with open positions, it is not testing the system that capital actually runs on.

The remediation signal is equally clear: state-persistent testing is a distinct test category. It requires a session primitive, replayed inputs, and explicit invariant assertions. It is additive to unit and integration testing, not a replacement.

The operational signal is that invariants asserted in testing should also be asserted in production. Same catalogue. Same thresholds. Same responses.