Can you trust an expert advisor’s backtest? The full report

An expert advisor is a trading robot; a backtest simulates a rule on historical prices. We tested the procedure of choosing a historical winning setting, then freezing it for later prices. No commercial expert advisor, vendor product or set file was tested. A backtest may accurately describe its fitting years while leaving later performance unanswered.

Returns compound completed trades over the whole requested window, not per year. Positions still open at the end are excluded. The headline figures are before trading costs. Buy-and-hold uses the stock’s last closing price in each window, so it treats the endpoint differently and is not matched to every risk characteristic of the strategy.

The locked design and its timestamp limit

The author recorded the plan as written 2026-08-15 at 18:39 UTC-07:00. It entered version history with the results at 19:09:58 UTC-07:00. No pre-run copy survives, so that history cannot independently prove the claimed order. This transcription is not an earlier timestamp.

The original plan contained qualitative terms such as “materially,” “strongly” and “large relative to” without numerical cutoffs. The table below preserves those limits. Failed predictions do not independently prove when a plan was written. Our earlier summary overstated the registered outcome; reading every clause corrected it to inconclusive.

Download the locked design

Every prediction and conditional disposition

Figures in this section come from a rerun on price bars cleaned by a rule set after the original results were known; the cleaning is not pre-registered (see Costs, missing histories and checks added later). Every grade is the same on the original bars.

The five numbered predictions and three conditional rules expand into thirteen clauses. The plan wording is included in the locked-design download. “Numerical condition met” never means that the registered test passed.

What the settings did: descriptive measurements

Figures in this section come from a rerun on price bars cleaned by a rule set after the original results were known; the cleaning is not pre-registered (see Costs, missing histories and checks added later). Every grade is the same on the original bars.

Returns compound completed trades over the whole requested window, not per year. Positions still open at the end are excluded. The headline figures are before trading costs. Buy-and-hold uses the stock’s last closing price in each window, so it treats the endpoint differently and is not matched to every risk characteristic of the strategy.

Keep both rising lines

These figures come from the cleaned rerun, which was not pre-registered. They are cross-instance means, not the preceding medians. Each row reads the later return of the winner selected on fitting prices. Both means rise; the smaller improvement is retained.

From one to 144 candidates, the fitting mean gains 52.10 points and the later mean gains 4.32 points. Their growth ratios are 7.502× and 1.504×: ratios of cross-instance means, not means of individual growth ratios.

The worked example and the whole surface

The closest-to-median rule stated in the lab’s results note picks MACROSS on MMM (3M), with 11 fitting trades. Its selected return is 28.88% and its typical setting is 0.86%, before costs. An example with eleven trades cannot represent the full experiment.

Its 4 one-step neighbors have median 13.92%; the gap is 14.96 points. Its later selected return is 9.03%. These extra example cells are descriptive and are not spoken in the episode; the full collection supplies the later comparison.

For the aggregate map from the cleaned rerun (not pre-registered), rank each fitting grid from 1 (best) to 16, then average each grid position across all 106 cases. The no-preference reference is 8.5. A flat rank map is not a zero-return surface.

All 16 of 16 grid positions won for at least one case; the most frequent position won for 20 of 106. Different positions won for different instances; this is not a registered pass.

Costs, missing histories and checks added later

The typical-setting fitting return falls from median 7.05% before costs to 5.01% with 0.05 of initial risk charged per completed round trip. This is the typical-setting slice, not the selected-winner headline.

The daily store includes delisted names on their historical dates, and the universe was selected as it stood at the start. That avoids selecting only present-day survivors, but ticker strings do not reliably track an instrument through renames or reuse. Of 60 selected symbols, 53 qualified. Six of the seven excluded symbols had no later bars; SBC had 73. MOT and VIA.B continued under new symbols, which this test did not follow. Seven exclusions do not mean seven companies died.

The original results existed before suspect prints were found. The cleaning rule was not pre-registered: it was recorded before the clean rerun, after those original results were known, and dropped 2 close-spike rows, clipped 3 opens and 17 wicks, across 22 field/row touches. The clean rerun provides this report’s headline figures. The rule uses a single 1.5 ratio; it cannot distinguish genuine extremes from bad prints, can clip real events (including a possibly real flash-crash low), and misses errors inside its threshold. “Clean” means processed by that rule, not certified correct prices.

The original raw-bar figures and the rerun with open-price clipping omitted were retained internally for comparison; the primary values here come from the complete cleaning rule. Reproducibility checks establish calculation agreement, not input accuracy.

Joined-history sensitivity — not pre-registered

In a check that was not pre-registered, five qualified symbols—SUNW, DELL, VRTS, GM and MXIM—crossed gaps longer than 180 days, affecting 10 of 106 rule-and-stock cases. In that check, which was not pre-registered, SUNW joined different companies under one symbol and accounts for 2 of those 106 cases, one per rule family. The other four were not established by that check to be different companies.

The following comparison was not pre-registered: excluding the ten long-gap cases leaves 96. All seven descriptive yes/no checks keep the same outcomes, including P1’s failure, but figures move. This does not repair the histories or re-grade the registered test.

Parameter-rank follow-ups — not pre-registered

In a check that was not pre-registered, the fitting-versus-later rank correlation across sixteen settings had median -0.01 over 106 cases.

In a check that was not pre-registered, the fitting winner’s later percentile averaged 48.00, against a no-information reference of 50.

In a check that was not pre-registered, the same setting won both windows in 15 of 106 cases (14.2%); its equal-chance reference was 6.2%, also a comparison that was not pre-registered. This observation remains visible without becoming a skill claim or a registered pass.

Positions left open — not pre-registered

In a check that was not pre-registered, 41 of 106 fitting windows ended with a position open at the selected setting; 17 of 106 later windows did so in that check that was not pre-registered.

In a check that was not pre-registered, valuing those positions at the last close, without cost and without reselecting settings, moved median fitting alpha from -5.59 to -5.95 percentage points. Both are negative. No registered figure or grade changes.

This later check used one convention: last close, no cost, no next-open exit. It re-read only the sixteen headline settings, not the wider grid, search curve, spike/plateau statistics or friction cells. Joined histories remain included. A last stored close may precede the calendar window’s end, and sale at that price was not verified. It does not establish a full account-value simulation or the behavior of a rule that closes positions at the endpoint.

In that check that was not pre-registered, 8 of the 17 later windows and 1 of the 41 fitting windows ending open had price histories that stopped before the requested period ended; holding lengths count bars, not calendar days.

Supporting comparison from another experiment

A separate opening-range day-trade study provides the episode’s supporting comparison, not B03’s headline. Its three target distances are only three choices, not every possible target. The quantity is mean return per trade in initial-risk units, after that study’s 0.20% round-trip friction; it is not annualized or compounded.

Every instance and downloadable cells

One instance is one rule family on one stock. The 106 instances share rules and instruments; they are not 106 independent markets. The full selected-setting ledger below includes both windows, their stock benchmarks, costs, trade counts and the later-added rank fields. The grid download holds all 1,696 fitting cells; the search download holds all 848 per-instance candidate-count summaries. The two-window grid entails 30,528 strategy runs, a workload count rather than evidence strength.

Column dictionary: Ledger rows and the three instance downloads are the cleaned rerun (not pre-registered); 14 of the 106 instances differ from the original run. is = fitting window; oos = later window; net = compounded completed-trade return before costs (%); alpha = strategy net minus the same stock’s buy-and-hold return (percentage points); friction = cost of 0.05 initial risk per round trip; head16 = typical or full-grid summary; sig = signal length in days; k_atr = stop multiple; r_mult = target multiple. Surface gap uses percentage points; ratio uses multiples and is blank when its neighboring denominator is nonpositive. Every not_pre_registered column is a later-added descriptive check, not pre-registered. Dates are observed first/last bars, not a promise of continuous coverage. Search CSV is_net/oos_net are within-case mean selected returns over the stated draws, before costs; the page’s search table then averages across cases.

All selected-setting instances All fitting parameter grids All search-count summaries Joined-history comparison Locked design

What would change our mind

A broad parameter surface with several useful neighboring settings, paired with settings publicly frozen before a later test, would give us a concrete new candidate. A rule using the same setting across instruments and timeframes also deserves examination. A new registered version of our own experiment should separate joined instrument histories first. A stronger new result cannot retroactively pass this registered test.

Sources and reproduction

This report follows the episode’s final script and amended findings through the completed-trade, cleaning and instrument-identity disclosures. Its numbers come from the retained clean-run instance cells and the separately labeled later checks, not from a review page. The downloads contain derived experiment outputs, not a redistribution of the underlying price feed.

Daily prices, the delisted-name history and the ticker reference file used for the renames come from Massive.com (formerly Polygon.io), pulled once on 2026-06-22 and split-adjusted as of that pull. The supporting opening-range study used one-minute bars from the same vendor. EODHD, the market-data affiliate linked in the video description, did not supply these measurements.

For this report, the original aggregation code was applied again to the saved instance rows: 600 of 600 aggregate fields and 197 of 197 parameter-map fields matched. This is reaggregation, not a fresh strategy run or an independent engine replication. The previously recorded engine comparison matched 1,024 of 1,024 backtests; that earlier test checks two implementations, not data quality.