Train / Eval Leakage Detector
Find evaluation records that also appear in training. Contaminated eval sets produce scores that look great and mean nothing. Compare on whole records, one field, the first user prompt, or the whole conversation — with case and whitespace normalisation so near-duplicates are caught too.
Compare
Which comparison mode to use
- Whole record — canonical comparison of the entire object, so key order and formatting differences are ignored. Catches copy-paste duplication. The strictest and fastest check.
- Full conversation text — flattens the conversation to
role: textlines before comparing. Catches records that are the same exchange wrapped in different metadata. - First user prompt — compares only the opening question. This is usually the mode that finds real contamination: the same question with a differently-worded answer is still a leaked test item.
- A specific field — compare on an id, a source URL, a question hash, or any dotted path such as
meta.source_id. Best when your records carry a natural identifier.
The normalisation toggles matter more than they look. With Ignore case on and whitespace collapsed,
"What is 2+2?" and "what is 2+2? " match — and they should.
Turn on Ignore punctuation for prose datasets where the same item was re-typed rather than copied.
What to do when you find leakage
Remove the contaminated records — but think about which side to remove from.
- Remove from evaluation when the training set is fixed or expensive to regenerate. Your eval set shrinks but becomes honest.
- Remove from training when the eval set is a curated benchmark you must report on unchanged.
- Re-split from scratch when leakage is widespread — it usually means the split was made after duplication, not before. Train / val / test split and k-fold split both support grouping so related records stay together.
The root cause is almost always duplicates in the source. Deduplicate before splitting and this problem mostly stops happening.
Privacy
Nothing is uploaded. The whole thing runs in this tab, in your own browser. That matters here more than on most tool sites — training data is usually the most sensitive file a team owns.
Frequently asked questions
It reports no leakage but I know there are duplicates.
You are probably comparing too strictly. Whole-record mode needs byte-identical data after canonicalisation; switch to First user prompt or Full conversation text, and turn on Ignore punctuation.
How large a file can it handle?
The comparison builds an index of the training set in memory, so it is fast — hundreds of thousands of records are fine. Very large files are limited by browser memory rather than by the algorithm.
Does it find near-duplicates, like paraphrases?
No. It compares normalised text, not meaning, so a paraphrased question will not match. Semantic deduplication needs embeddings and a similarity threshold, which is not something to do in a browser tab.
Can I compare three sets?
Two at a time. Run train↔val, then train↔test, then val↔test — all three pairs matter, and val↔test overlap is the one people forget.