JSONL Near-Duplicate Detector
Exact deduplication misses the duplicates that matter. Two records with the same answer reworded, the same article with a different byline, the same prompt with a trailing space — byte comparison sees two distinct records, and a model trained on them sees the same example twice. This finds them with MinHash and locality-sensitive hashing, reports the similar pairs, and can hand back a file with one record per cluster.
Detect near-duplicates
How it works, and why it is fast
Each record's text is lowercased, stripped of punctuation and cut into overlapping n-grams — with the default shingle size of 5, the sentence becomes every run of five consecutive words. Two records are compared by the Jaccard similarity of their shingle sets: shared n-grams over total distinct n-grams.
Comparing every pair would be quadratic and hopeless past a few thousand records. Instead each set is reduced to a 64-value MinHash signature — a fingerprint with the property that the chance two signatures agree at a position equals their Jaccard similarity — and the signature is cut into 16 bands. Records sharing any whole band become candidate pairs, and only those candidates get an exact similarity computed. That is standard LSH, and it makes the work proportional to the number of plausible pairs rather than to the square of the file. The status bar reports how many candidate comparisons were actually performed.
Choosing a threshold and a shingle size
0.8 with 5-word shingles is a sensible default for prose: it catches paraphrases and boilerplate reuse while leaving genuinely different documents alone. Lower the threshold toward 0.6 to catch looser reformulations and expect more false positives. Raise it to 0.95 when you only want near-identical text.
Shorter shingles (2–3 words) make short records comparable at all — with 5-word shingles, a four-word record produces a single shingle and matches almost nothing. Longer shingles (7–10) are stricter and are the right choice for long documents where shared common phrasing would otherwise inflate the score. If your records are single sentences, drop the shingle size to 3.
The banding is fixed at 16 bands of 4 rows, which gives a soft detection curve centred near 0.7 — pairs well above the threshold are found essentially always, and pairs just below it are occasionally missed. This is a probabilistic method: it is the only kind that scales, and it is what the large public deduplication pipelines use.
What gets read as the record's text
Name a field to be explicit. Left blank, the tool looks for a messages array and concatenates its
content, then falls back to the first of text, content, completion,
output, response, prompt, instruction. For chat data,
naming the field you actually care about — usually the assistant turn — gives sharper results than the default,
which mixes prompt and answer together.
Privacy
Nothing is uploaded. The whole thing runs in this tab, in your own browser. That matters here more than on most tool sites — training data is usually the most sensitive file a team owns.
Frequently asked questions
Which record survives deduplication?
The first one in file order within each cluster. If you want a specific one to win — the longest, the newest — sort the file first so that record comes first, then deduplicate.
Is this the same as the leakage detector?
Related but not the same. Leakage detector compares two files — typically train against test — to find contamination across a split. This works within one file to find internal redundancy. Run this before splitting and that one after.
Why does the report show fewer pairs than records in clusters?
Because clusters are transitive and pairs are not. If A matches B and B matches C, all three end up in one cluster even though A and C were never compared or did not pass the threshold themselves. The cluster count is what determines how many records deduplication removes.
How large a file can it handle?
The shingle sets for every record are held in memory at once, so text volume matters more than record count. Twenty thousand records is the default cap and works comfortably; raise it if your records are short and lower it if they are documents. Anything larger belongs in a batch pipeline, where the same MinHash approach is what tools like datasketch implement.