Backtesting and Strategy-Simulation Infrastructure for Trading Systems

Backtesting infrastructure has one job that sounds simple and is not: replay history exactly as a strategy would have seen it, one decision at a time, without ever letting the strategy peek at information that did not exist yet. Almost every serious bug in this space is a variant of that same failure, wearing a different costume.

Look-ahead bias is the obvious one — joining a feature table on a timestamp that was actually revised days later, or using an end-of-day close to make a decision that, in reality, could only be made intraday. Survivorship bias is subtler: if your instrument universe is built from today's index membership, you have silently deleted every company that was delisted, merged, or dropped, and the simulation is now testing against a universe that only exists in hindsight. Data leakage through feature engineering is subtler still — a rolling z-score computed over a window that includes future bars, or a categorical encoding fit on the full dataset instead of only the data available at decision time.

The second hard constraint is nonstationarity. Market microstructure, fee schedules, tick sizes, and even instrument definitions change over years of history. A simulation engine that treats history as a single homogeneous dataset will quietly misprice execution in regimes where the rules were different. The architecture has to treat 'what the market looked like' as a first-class, time-varying input, not a static reference table bolted on afterward.

The third constraint is one every fintech system eventually hits: reproducibility. A backtest run on Tuesday and rerun on Friday, against the same code and the same date range, must produce bit-identical results. Without that guarantee, every optimization, every parameter sweep, and every regression test becomes unfalsifiable — you can never tell if a change in output came from your code or from nondeterminism in the pipeline.

The foundation is a bitemporal data model — every fact is stored with both a valid-time (when it was true in the world) and a transaction-time (when the system learned it). A price revision, a restated earnings figure, an index reconstitution — none of these overwrite history. They are appended as new rows with a later transaction-time, and the original row stays exactly as it was. A backtest run 'as of' a given transaction-time sees only what was actually knowable at that moment; a backtest run 'as of today' sees the fully restated dataset. Both are valid queries against the same store, and the difference between them is precisely the class of bug this whole exercise exists to prevent.

Corporate actions — splits, dividends, symbol changes, delistings — live in their own versioned tables and are applied as adjustment functions at query time rather than baked into a single 'adjusted close' column. Baking in adjustments early is a common shortcut that becomes a liability the moment you need both adjusted and unadjusted views, or need to explain to an auditor exactly how a historical price was derived.

For storage, columnar formats (Parquet-style, partitioned by instrument and date) dominate for the bulk of historical market data because backtests are overwhelmingly scan-and-filter workloads — read a date range and a column subset, not point lookups. A separate time-series store, or an in-memory columnar cache in front of the same partitions, handles the tighter read patterns of interactive research. The two layers share the same underlying partition layout so a query written against the research cache and a query written against the full historical store return identical results, just at different latencies.

Two architectural styles show up repeatedly, and a serious platform usually needs both. An event-driven engine processes a strictly ordered stream of market events — quotes, trades, order-book deltas — and delivers them to the strategy one at a time through the same interface it would use in production. This is the higher-fidelity mode: it can model queue position, partial fills, and realistic order-to-fill latency, and because the strategy code path is identical to production, it doubles as an integration test for the live execution logic. The cost is throughput — event-driven replay of years of tick data for a large parameter sweep is expensive.

A vectorized engine instead operates on whole arrays of bars at once, computing signals and simulated fills as batch operations. It is dramatically faster and is the right tool for early-stage research and broad parameter sweeps, but it necessarily approximates execution — it cannot easily represent 'my order sat in the queue behind three others and only partially filled.' The trade-off is well understood and the right answer is a tiered pipeline: vectorized search over a wide parameter space to find promising regions, then event-driven replay of the shortlisted candidates for a higher-fidelity check before anything goes further.

Execution modeling is where a backtest either earns trust or loses it. A fill model needs, at minimum, a slippage function tied to order size relative to observed liquidity, a latency model between signal and order arrival, and a fee/rebate schedule that matches the venue and date being simulated — fee schedules change, and a static fee constant across ten years of history is its own silent bias. None of this should be presented as a prediction of live behavior; it is an explicit, inspectable assumption that the platform should make visible in every result, not hide inside a black box.

Because the failure modes here are silent by nature — a leaky backtest doesn't crash, it just produces numbers that look plausible — correctness has to be enforced structurally, not just tested for. A dedicated leak-detection pass can instrument the data-access layer so that every read is checked against the simulated 'current time' of the engine; a strategy that reaches even one bar past that boundary fails the run immediately rather than producing a silently optimistic result. This is a fail-closed design: an ambiguous or unverifiable data access should halt the run, not degrade gracefully into a wrong answer.

Walk-forward structure is enforced at the harness level rather than trusted to strategy authors: the platform partitions history into rolling train/validate windows and refuses to let a validation window's data influence anything computed for the training window ahead of it. This is deliberately inconvenient by design — it should be harder to write a leaky backtest than a correct one.

Determinism is treated as a build property, not an afterthought. Every run is pinned to a specific data-store transaction-time snapshot, a specific code commit, and a specific seed for any stochastic component (randomized fill assumptions, resampling, etc.), and the full set is hashed into a run identifier. Two runs with the same identifier are required to produce byte-identical output artifacts; if they don't, that is treated as a platform bug, not noise. This is what makes it possible to trust a parameter sweep across thousands of runs — each one is independently reproducible and auditable after the fact, which matters as much for internal debugging as it would for any external review.

Parameter sweeps and multi-strategy research are embarrassingly parallel at the level of a single backtest run, which is the easy part — the hard part is making each run cheap enough that a sweep of thousands is tractable. That pushes toward a distributed job model: a scheduler fans out independent (strategy, parameter set, date range) jobs across a compute cluster, each job reads from the shared columnar store, and results land in an append-only results table keyed by the run identifier described above. Idempotency matters here as much as it does in any payments pipeline — a retried or re-submitted job with the same run identifier must be a safe no-op, not a duplicate row.

Observability is intentionally boring: every run emits structured logs of which data partitions it touched, what the fill model assumed, and how long each phase took, because the questions that come up months later are almost always 'what data did this see' and 'why did this get slower,' not 'what was the average latency across all runs.' A run's provenance metadata is worth more than an aggregate dashboard.

The trade-offs worth naming plainly: we would not build a single monolithic engine that tries to be both the fastest vectorized researcher and the most faithful order-book simulator — that engine ends up mediocre at both. We would not adjust prices in place at ingestion time, even though it's simpler, because it forecloses point-in-time correctness later. And we would not treat a passing backtest as evidence of anything beyond 'the strategy behaved as coded against this specific, explicitly-assumed execution model' — the fill model, the fee schedule, and the universe construction are all engineering decisions with real consequences, and the platform's job is to make every one of them visible, versioned, and swappable, not to produce a single number that hides them.