How to Verify Historical Tick Data Integrity

You download 40 million ticks.

The file opens.

There are no obvious errors.

So the data must be good.

Right?

Not necessarily.

Data integrity is not the same thing as “the file exists”.

For serious backtesting, you want evidence that the history is internally coherent and that suspicious periods have been identified rather than silently accepted.

Start with the simplest checks

Before doing anything clever, verify the basics.

File readability

Can every row be parsed?

Malformed or truncated rows should not simply disappear without being reported.

Timestamp ordering

Tick history should move forward through time.

Out-of-order records deserve investigation.

Duplicate behaviour

Exact duplicate records can arise legitimately in some workflows, but unexpected duplication can also indicate a merge or download problem.

Detect it.

Measure it.

Do not guess.

Price sanity

For a normal quote feed, Bid and Ask should be numerically valid.

You should look for:

• zero or missing prices,
• impossible negative values,
• obviously corrupted spikes,
• Ask below Bid where that is not expected.

Then move from rows to time

A clean file can still contain missing history.

That makes coverage analysis essential.

Instead of asking:

“How many rows do I have?”

ask:

“Which periods can I account for?”

Segment the dataset by:

• year,
• month,
• day,
• hour,
• minute where appropriate.

Now prolonged periods without ticks become visible.

But a gap is not automatically missing data

This distinction is crucial.

Markets close.

Holidays exist.

Some instruments have scheduled session breaks.

Illiquid periods can genuinely contain few updates.

So an integrity checker should not blindly label every empty hour as corruption.

It needs context.

A missing hour on Saturday in spot FX is very different from a missing hour during an active London session on a normal Tuesday.

Reconstruct candles from the ticks

One of the strongest consistency tests is to aggregate the tick stream into bars.

From the appropriate price side, reconstruct:

• Open,
• High,
• Low,
• Close,
• tick count.

Now inspect whether the reconstructed timeline behaves as expected.

This is conceptually similar to the consistency work MetaTrader itself performs when using minute bars alongside real tick history in its Strategy Tester.

Pay special attention to time-zone boundaries

Data that is perfectly ordered can still be mapped to the wrong clock.

Check:

• day boundaries,
• weekend opens,
• weekend closes,
• known daylight-saving transition dates.

A one-hour time error can leave every individual price untouched while changing the candles produced from those prices.

That is why timestamp validation belongs inside integrity checking.

Look for statistical anomalies — carefully

Abrupt changes in:

• tick frequency,
• average spread,
• price jumps,
• zero-volume patterns,
• missing hours

can identify suspicious regions.

But anomaly detection should generate questions, not automatically delete data.

A central-bank announcement really can create an extreme move.

Christmas really can create unusual liquidity.

The job of an integrity system is to distinguish unusual from unexplained.

Retry what can be verified

Network failures happen.

A robust pipeline should not simply write “missing” into a log and move on forever.

If the source supports deterministic retrieval, suspicious or incomplete intervals can be fetched again and compared.

That makes the integrity process repeatable.

Record what was verified

A useful data-quality report should tell you more than “PASS”.

Ideally it records:

• source,
• instrument,
• requested period,
• actual coverage,
• tick count,
• malformed rows,
• duplicate observations,
• suspicious gaps,
• intervals rechecked,
• intervals still unverified,
• integrity timestamp,
• file checksum or equivalent provenance.

That turns a pile of historical files into a traceable dataset.

“Unverified” is a valid result

Sometimes a source cannot prove an interval conclusively.

Do not convert uncertainty into false confidence.

A transparent dataset can say:

verified

missing

unverified

That is much more useful than a quality percentage with no explanation of how it was produced.

Backtesting starts before the Strategy Tester

A backtest is a calculation performed on assumptions and data.

If the data layer has not been inspected, part of the experiment remains unknown.

That is why integrity checking is not housekeeping.

It is part of strategy research.