An AI Agent’s Backtest Must Be Reproducible

An AI agent runs a strategy test and reports lower drawdown, more trades and a better result. That sounds like progress. But can anyone verify that the strategy itself caused the change?

One run may have used a different date range. Another may have applied a different fee. The market history could have changed while the slippage model went unrecorded. Even a small change in execution assumptions can reshape the result although the strategy code looks almost identical.

A good backtest is therefore not a single number or a polished equity curve. It is an experiment that another person—or the same agent—can repeat with the same inputs and obtain a comparable result.

The same idea can produce different results

Consider a simple hypothesis: buy when price moves back above a moving average, then exit on a stop or a reverse signal.

The agent runs two tests.

The first uses candles from one exchange, market-order execution and a fixed fee. The second uses history from another venue, limit orders and a more forgiving slippage assumption. The strategy name is unchanged and the parameters look familiar, but these are now two different experiments.

Comparing only their returns is meaningless. The first question should be: what stayed constant, and what changed?

What must be stored with every result

A reproducible backtest starts with a run manifest, not a performance metric.

1. The exact strategy version

“Mean reversion v7” is not enough. The record needs the source code—or an unambiguous version-control reference—plus parameters and every related file.

If the agent changes a single line after seeing the result, that is a new version. The old result cannot be attributed to the new code.

2. The market data

Record at least:

venue and instrument;
spot or futures;
timeframe;
start and end dates;
source and history version;
gaps and the method used to handle them.

BTCUSDT from two exchanges is not the same dataset. Venues differ in candles, liquidity, funding and sometimes even price construction.

3. The execution model

A strategy signal is not yet a trade. Between the two sit:

order type;
fees;
slippage;
latency;
available liquidity;
partial fills;
funding for perpetual futures;
mark-price and liquidation rules.

If these assumptions are not recorded, the result cannot be reproduced. The agent may have followed the request perfectly and still produced an “improvement” only because the second test was more generous than the first.

4. The experiment settings

Store initial capital, position rules, leverage, strategy parameters, train/test windows, walk-forward design and the search budget.

If the process contains randomness, record the seed. If the agent tested dozens of variants, preserve the number of attempts rather than only the winner. Otherwise, one fortunate run can be mistaken for a robust pattern.

5. The environment version

The strategy is only one part of the result. The backtester version, simulation rules, compiler and connected components also matter.

After an update, an old experiment should either be replayable or clearly tied to the version in which it ran.

6. The complete result artifact

Total return does not explain how a strategy behaved. A useful run record also includes:

the trade list;
equity curve;
maximum drawdown;
fees and funding;
errors and rejected actions;
execution log;
timestamp and run identifier.

This makes it possible to see not only what changed, but why.

How an AI research agent should work

A useful agent does more than search for a better metric. It keeps an experiment ledger.

Before each run, the agent states a hypothesis—for example, “a volatility filter should remove entries during low-activity regimes.” It changes only the declared component, keeps data and execution assumptions fixed, runs the test and compares the outputs against criteria chosen in advance.

Afterward, it should report:

what changed;
what stayed constant;
which data range was used;
how many variants were tested;
where the code and result artifacts are stored;
which limitations prevent the result from being treated as evidence for live trading.

That report is less dramatic than “I improved the strategy by 37%.” It is also much easier to verify.

Where MCP helps

MCP lets an agent interact with a testing environment through structured capabilities instead of guessing application state from screenshots or manually copying numbers out of a table.

XTester publicly describes capability classes for environment discovery, project work, compilation, market history, backtesting, results, run comparison and research. Exact tool names and schemas must come from the catalog exported from the matching verified release; they should not be invented before that catalog is published.

The protocol does not make research reproducible by itself. That remains a workflow decision. It does, however, make it possible to build a cycle in which the agent reads the environment first, records inputs, performs a bounded change and returns inspectable artifacts.

A minimum checklist before trusting a comparison

Before comparing two backtests, verify all six items:

the code version is fixed;
the dataset and date range match;
the execution model matches;
parameters and experiment budget are recorded;
the test-environment version is identified;
trades, metrics and the run log are available.

If any item is unknown, the result may still be a useful clue. It is not yet evidence of an improvement.

AI can accelerate the production and testing of hypotheses. Reproducibility keeps that speed from turning into a stream of attractive but incomparable reports.

XTester is a development and historical testing environment for trading strategies. Backtests do not guarantee future returns and are not investment advice.