Backtesting Without Fooling Yourself

The backtest is the most powerful tool a quant has - and the easiest way to lie to yourself ever invented. A handful of subtle, almost-invisible mistakes can turn a worthless or even losing strategy into a backtest so beautiful you'll bet real money on it. We've touched the dangers before (multiple testing in Chapter 13, out-of-sample in Chapter 27); this chapter confronts the specific biases that corrupt backtests head-on, because surviving them is the entire difference between a quant who lasts and one who blows up on their first live deployment.

Look-ahead bias

The deadliest bug is look-ahead bias - using information in your backtest that you wouldn't actually have had at the time. It's usually a one-line mistake: forgetting to lag your signal, so your strategy "decides" today's trade using today's closing price, which it couldn't possibly know until the day is over. Watch what that single error does:

# Look-ahead bias: forgetting to lag the signal by one bar fakes a brilliant edge.
import os
from datetime import datetime

import numpy as np
from openalgo import api

client = api(
    api_key=os.getenv("OPENALGO_API_KEY", "your_api_key_here"),
    host=os.getenv("OPENALGO_HOST", "http://127.0.0.1:5000"),
)

end = datetime.now().strftime("%Y-%m-%d")
c = client.history(symbol="NIFTY", exchange="NSE_INDEX", interval="D",
                   start_date="2021-01-01", end_date=end)["close"]
r = c.pct_change()

sma = c.rolling(20).mean()
signal = np.sign(c - sma)        # +1 above the average, -1 below (uses today's close)


def sharpe(x):
    x = x.dropna()
    return x.mean() / x.std() * np.sqrt(252)


cheat = sharpe(signal * r)          # BUG: trade today's return with today's signal
honest = sharpe(signal.shift(1) * r)  # correct: decide on yesterday's close, trade today

print(f"With look-ahead (signal * today's return) : Sharpe {cheat:+.2f}   <- looks amazing")
print(f"Correctly lagged (signal.shift(1))        : Sharpe {honest:+.2f}   <- the truth")
print(f"\nThe entire 'edge' was a one-bar indexing error. Always lag your signal before the return.")

With look-ahead (signal * today's return) : Sharpe +4.51   <- looks amazing
Correctly lagged (signal.shift(1))        : Sharpe +0.16   <- the truth

The entire 'edge' was a one-bar indexing error. Always lag your signal before the return.

A Sharpe of 4.51 - a number that would make any fund salivate - collapses to 0.16 the instant you lag the signal correctly. The "edge" was never real; it was the strategy peeking at the answer. And the equity curves make the fiction unmissable:

Copy

# The look-ahead equity curve rockets; the honest one crawls. Same code, one-bar bug.
import os
from datetime import datetime
from pathlib import Path

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
from openalgo import api

client = api(
    api_key=os.getenv("OPENALGO_API_KEY", "your_api_key_here"),
    host=os.getenv("OPENALGO_HOST", "http://127.0.0.1:5000"),
)

end = datetime.now().strftime("%Y-%m-%d")
c = client.history(symbol="NIFTY", exchange="NSE_INDEX", interval="D",
                   start_date="2021-01-01", end_date=end)["close"]
r = c.pct_change()
signal = np.sign(c - c.rolling(20).mean())

cheat = (1 + (signal * r).fillna(0)).cumprod()
honest = (1 + (signal.shift(1) * r).fillna(0)).cumprod()

sns.set_theme(style="whitegrid")
fig, ax = plt.subplots(figsize=(8, 4.5))
ax.plot(cheat.index, cheat, color="#dc2626", lw=1.8, label="with look-ahead (a lie)")
ax.plot(honest.index, honest, color="#16a34a", lw=1.8, label="correctly lagged (the truth)")
ax.set_title("Same strategy, one-bar bug - the look-ahead curve is fiction")
ax.set_ylabel("Growth of 1")
ax.legend()

out = Path(__file__).with_suffix(".png")
plt.savefig(out, dpi=110, bbox_inches="tight")
print(f"Look-ahead final {cheat.iloc[-1]:.1f}x vs honest {honest.iloc[-1]:.2f}x. Saved {out.name}")

Live output

Look-ahead final 25.2x vs honest 1.07x. Saved 02_lookahead_equity.png

The red look-ahead curve rockets to 25x; the honest green one crawls to 1.07x. Same code, one missing .shift(1). This is why experienced quants are paranoid about timing: every signal must be computable strictly from information available before the bar it trades. A Sharpe that looks too good to be true almost always is - and look-ahead is the usual culprit.

Survivorship bias

The second trap is survivorship bias - testing your strategy only on the stocks that still exist today. Build a universe from today's Nifty 500 and backtest it over ten years, and you've quietly excluded every company that went bankrupt, got delisted, or collapsed along the way. You're testing only on the survivors - the winners - which makes almost any strategy look great. The fix is point-in-time data: at each historical date, use the universe as it was then, including the names that later died.

Data snooping and overfitting

The third is data snooping - the multiple-testing mirage of Chapter 13. Try enough parameters, indicators and universes, keep the best, and you've manufactured a fluke. Every backtest you've ever seen reported is the survivor of a search you weren't shown. The defences are familiar: honest out-of-sample testing (Chapter 27), counting your trials, and demanding an economic reason.

Walk-forward analysis

So how do you backtest rigorously? The gold standard is walk-forward analysis. Instead of one train/test split, you roll the window forward again and again - fit on a stretch of history, test on the next unseen slice, then slide forward and repeat:

The strategy is always judged on data it never saw during fitting, and across many such tests - so a parameter that only worked by luck in one period gets exposed in the others. Walk-forward mimics how you'd actually trade: periodically re-fitting on recent history and trading the next stretch. It's the closest a backtest gets to honesty.

The deflated Sharpe ratio

Even a clean walk-forward Sharpe needs a haircut for how hard you searched. The deflated Sharpe ratio adjusts your reported Sharpe downward based on the number of strategies you tried - because the more you test, the higher the best one will be by luck alone (Chapter 13). A Sharpe of 2 found after a thousand trials is far less impressive than a Sharpe of 1 found on your first honest attempt. Always discount for the size of your search.

Realistic costs

Finally, a frictionless backtest is a fantasy. Every test must include realistic costs - the full STT-heavy stack of Chapter 4, plus the spread and market impact of Chapter 7. Many a strategy with a glorious gross return is a steady loser after costs, especially the high-turnover ones. Backtest net, always, or you're trading a strategy that only existed in a world without friction.

The backtester's checklist

Before believing any backtest, confirm:

• Is every signal lagged so it uses only past information?
• Is the universe point-in-time (no survivorship)?
• Is it validated walk-forward / out-of-sample?
• Is the Sharpe deflated for the number of trials?
• Are realistic costs (STT, spread, impact) included?

Try it yourself

• Deliberately introduce other look-ahead bugs - normalise returns using the full-period mean, or use tomorrow's high as a signal. How absurd does the Sharpe get?
• Add costs to the honest version of the strategy. Does its modest 0.16 Sharpe survive friction, or turn negative?
• Implement a simple two-fold walk-forward on any strategy. Does the out-of-sample performance match the in-sample, or fall off a cliff?

Recap

• Look-ahead bias - using information you wouldn't have had - is the deadliest bug: one missing lag faked a Sharpe of 4.51 (truly 0.16) and a 25x curve.
• Survivorship bias inflates results by testing only on stocks that survived; fix it with point-in-time universes.
• Data snooping manufactures flukes by searching many ideas - defend with out-of-sample testing and honest trial counting.
• Walk-forward analysis - roll the train/test window forward repeatedly - is the rigorous standard; the deflated Sharpe discounts for how hard you searched.
• Always backtest with realistic costs - a frictionless backtest is a fantasy that dies on contact with the market.

A clean backtest is hard enough; bring machine learning into the mix and the ways to fool yourself multiply. Next we look at where ML genuinely helps a quant, and the subtle leaks that make it dangerous.