JSONL Quality Filter
The junk in a training set has a shape, and it is measurable. Empty answers, one-word replies, text that loops on itself, ALL-CAPS shouting, records that are 40% punctuation because a scraper caught markup. This applies the standard heuristics, tells you which rule each rejected record broke, and hands back either the survivors or the rejects — so you can read what you are about to throw away before you throw it away.
Filter
What each rule catches
Minimum characters and words remove the empty and near-empty answers that dominate scraped
datasets — ok, yes, a single emoji. These two alone usually account for most of what a
first pass rejects.
Repeated 3-grams catch degenerate generation: text that restates the same clause, a list that repeats an item, an answer that loops until the token limit. The measure is the share of overlapping three-word windows that have already appeared in the same record, so ordinary prose sits well under 0.1 and a looping record climbs past 0.5. It only applies to records of ten words or more, since short text has too few windows to judge.
Uppercase share is computed over letters only and applies from twenty letters up, so acronyms and short labels are not penalised. Symbol share counts characters that are neither letters, digits nor whitespace — high values mean markup, code fragments in a prose dataset, or a mojibake encoding problem, which the encoding fixer handles better than deletion.
Read the rejects first
Every heuristic here is a proxy, and a proxy tuned too tight deletes good data. Set the output to records that
failed before you trust any threshold: each rejected record carries a __rejected array naming the
rules it broke and the measured value, so you can see whether "too short (18 chars)" was junk or a perfectly good
terse answer. The status bar tallies rejections by rule, which tells you at a glance which threshold is doing the
work.
Everything, rejects annotated is the mode for auditing: nothing is removed, and the annotation travels with the record so you can filter it downstream or hand the file to a reviewer.
Privacy
Nothing is uploaded. The whole thing runs in this tab, in your own browser. That matters here more than on most tool sites — training data is usually the most sensitive file a team owns.
Frequently asked questions
Should I set a maximum length?
Only if you know your context limit and want the file to fit it. Max chars is 0 — off — by default, because truncating by rejection loses whole records where counting tokens and splitting would keep them. Use this to drop pathological outliers, not to fit a budget.
Does it judge the prompt or the answer?
The answer, by default — for chat records it takes the last assistant message. Prompt quality is usually a separate question with different thresholds, so name the prompt field explicitly and run a second pass if you need both.
Why is a record with no text rejected as "too short"?
Because an empty extraction is indistinguishable from an empty answer at this level. If many records report zero characters, the field auto-detection did not find your text — name the field explicitly.
Is this enough to clean a dataset?
No. It removes the mechanically bad records. It cannot tell you whether an answer is correct, safe or on-topic, and it does not touch duplication — pair it with near-duplicate detection, which typically removes more records than the quality rules do.