An overfit check meets coin-flip trading strategies

Source video: GPT-6 Astra Built 200 Python Strategies. Here’s the Winner (300%), Algo-trading with Saleh, by Saleh Mir. The percentage in that video title is his reported figure, not ours. He maintains Jesse, makes the tool available, shows losing variations and warns that a check cannot guarantee future returns. We do not know who operated that check, its exact settings or whether other held-out testing happened elsewhere.

We tested an instrument: Jesse’s Monte Carlo candles. It reruns a fixed strategy on rebuilt price histories. It does not repeat the original strategy search inside each rebuilt history. Rebuilt historical prices and data withheld from strategy selection answer different questions.

Pre-registered goalposts and dated changes

The initial plan survives in commit 7a80a14, authored 2026-09-28T23:40:40-07:00, before the recorded runs. It predicted an eleven-year flag and the same class across windows. Its original data object and parity test were superseded through dated amendments. Cleaning was adopted after a diagnostic revealed a winner; it was not pre-registered before that diagnostic. The active experiment used the repaired rows in Jesse’s own backtester. Repository dates are not external timestamp attestation.

Deliberately NOT tested: the source strategy, its unseen code, its actual Monte Carlo settings, a causal isolation of window length, and the other videos in the series. Unsettled source risk-per-trade and holding-time claims are not used. SPY is our substitute for the source’s AAPL; matching a two-year length is not reconstructing the source experiment.

Our predictions, including both failures

Our two predictions together implied a flag on two years. That flag occurred, but it does not rescue the separate same-class prediction. The earlier unqualified hunch that this tool could not catch selected coin flips was too broad: the two-year result contradicts it. The specific primary-window rule and our editorial decision retain a standalone episode with both outcomes; no numerical result was reclassified. Proposed mechanisms remain inferences.

What the two searches and checks returned

All own strategy returns below are total backtest returns over the stated window, before trading fees, not annual rates or forecasts. They are from the amended design on repaired price data. The cleaning decision followed a diagnostic; it was not pre-registered before that diagnostic. Buy-and-hold is the same fund’s price change, not a risk- or leverage-matched portfolio and not dividend-inclusive total return.

On the eleven-year search, the selected winner returned +322.83% across 353 trades. Across that same 2,048-configuration search, the median was -63.33% and 78.5% lost money. Selection makes this winner exceptional within its own searched history; the random-entry code establishes why its signal contains no price information.

On the later window, 2015-01-01 through 2026-06-22, that same frozen eleven-year winner returned +93.27%, while holding SPY returned +262.1%. It made money, and it earned much less than holding the asset. A later profit does not make the entry signal informed. The return comparison is separate from the Monte Carlo classification.

The two-year search selected its own configuration. We did not simply shorten the eleven-year winner’s chart. Both the window and winner changed, so this comparison does not isolate which caused the classification difference.

Controls that can fire, and controls that should stay quiet

On eleven years, full foresight was flagged; 0 of 10 unselected random configurations were flagged. On two years, full foresight was flagged; 0 of 10 unselected configurations were flagged, and 1 was grey. Grey is not a flag. These controls show useful discrimination in these constructions, not universal reliability.

The pre-registered conditional partial-foresight arm ran because the eleven-year winner did not flag. Preset steps overshot its target: on eleven years, partial foresight returned +470.30%, about 1.46 times the eleven-year winner’s return. It also passed on eleven years; the two-year selected coin-flip winner was flagged. We did not retune the dose to an exact match. This rules out a neat claim that the instrument catches foresight but misses selection.

Repairs, deviations and limits

Price repair was adopted after the diagnostic and was not pre-registered before it. The final rule replaced 2 bad closes and 1 bad open, clipped 6 highs and 7 lows, without dropping rows. It uses a ten-percent neighbor/body threshold; the bad-open clause was added after inspecting the raw rows. Such a rule can alter genuine extremes and cannot certify prices correct. The old dirty-data winner supplies no evidence for this episode.

Cross-engine parity then failed structurally: fills at an hour’s close and the next hour’s open can resolve stops/targets differently. The dated third amendment selected afresh inside Jesse and replaced parity with repeatability checks. We do not claim that the two engines agree. The initial wrong-unit schedule and failed/cached partial-foresight attempts were excluded; no preferred failed attempt was selected for publication.

A corrective rerun, not a separately pre-registered arm, repaired the full-foresight construction. The initial eleven-year control kept 93 in-run winning entry hours; the literal judged-alone rule kept 97. They agreed for 349 of 353 trade signs. Both eleven-year controls flagged. The two-year control agreed under both readings. Both versions remain in the arm ledger, with the correction labeled.

The eleven-year selected-winner run returned 187 of 200 requested scenarios; 13 failed scenarios were dropped by the tool. The two-year winner returned 200 of 200. A diagnostic after results, not a pre-registered causal experiment, traced failures to rebuilt paths reaching the price floor and invalid stops. The window and winner changed together; price wander is a clue, not an isolated cause. Dropped-path bounds were later descriptive checks, not a new class rule.

Fixed seeds, reduced control scenario counts, zero fees, long-only entries, hourly SPY prices and leverage conventions are ours. The source prompt included trades in both directions and a slower chart. The source demonstration does not describe held-out data on camera; that observation is not proof that none existed elsewhere. Its strategy’s overfit status remains unknown from this experiment.

Every arm and reproducible downloads

The ledger contains every completed arm, including the corrective full-foresight rerun. N is valid/requested rebuilt histories, not configurations. Original return is total percent before fees; Sharpe is the tool’s backtest statistic. z measures the original’s position between median and upper line; p_value is the returned fraction of scenarios at least as high as the original, not a posterior probability of overfitting. Only Sharpe determines class.

Download dictionary: arm CSV columns are instance, arm, window, valid, requested, z, cls, original, median, best_5, worst_5, p_value, original_return_pct, original_trades, evidence_class, source. Search CSV has window, cell, seed, p (entry offer rate), k (stop ATR multiple), net (whole-window return percent), total (trades), win_rate (fraction), max_dd (percent), sharpe, trades_sha256 and seconds (recorded runtime). Scenario CSV has arm/window, i (scenario index), sharpe, net (percent), mdd (percent), wr (win fraction), n (trades), calmar and min_entry (lowest entry price). Blank means unavailable, never zero. The downloads contain 4,096 search rows and 1,676 saved scenarios; underlying vendor price bars are not redistributed.

What would change our mind

1. Publication of the source strategy and settings, OR separately stated scenario count, pipeline or batch size: either independently earns a rerun with that specification.
2. Evidence the source used the Gaussian pipeline: rerun that method.
3. A Jesse release changing the check or defaults: rerun the primary test.
4. A defect in our code, rebuilt trade list or method reading: retain affected results as EXCLUDED, fix and rerun, even if the class stays unchanged.
5. A rerun that flags the eleven-year winner while unselected controls stay specific changes our primary conclusion; the two-year winner already flagged, so the older universal hunch stays retired.
6. An earlier winner retested on fresh data in another series episode would require rewriting the unmeasured secondary expectation. We made no corpus-wide finding here.

Two inherited conditions stay dormant because their premises are excluded here: a published source trade count in the hundreds defeats a low-count premise; a comparable pre-registered study ranking model class among the top two factors defeats the least-important-model premise. Nicer seeds, revised dates or deleting odd scenarios do not license choosing a preferred result.

Data, provenance and reproduction

Data vendor: Massive.com (formerly Polygon.io), Stocks Advanced, from the local SPY one-minute store pulled 2026-06-22. The store manifest says split-adjusted, not dividend-adjusted; hourly prices and both SPY buy-and-hold comparisons derive from these rows. This is one fund, not a claim of a survivorship-free strategy universe. EODHD supplied no data to this experiment. Affiliate links support tools, not a result, favorable verdict or place in the evidence; we never take money from anyone whose claims we test.

This packaging pass re-executed the production classification function on saved confidence tables: 26 of 26 classes agree. It reproduced 468 of 468 percentiles from six saved scenario metrics and 14 of 14 winner-selection/summary fields from the two complete searches. A reversed-class control failed first. This is reaggregation, not rerunning strategies or independently replicating the engine. Annual-return scenario values were not saved and are not freshly checked. The design download lists source hashes.