JSONL for Machine Learning

Use case · updated 28 July 2026 · overview · spec · format comparison · examples · best practices · troubleshooting · FAQ · glossary

Every major fine-tuning API takes JSONL, because a training set is exactly what the format models: independent records, appendable, streamable, and reportable by line number when one is wrong. This is the order of operations that avoids paying for a failed training run.

Why every provider chose this format

OpenAI, Anthropic, Google, Mistral, Together, Hugging Face — all of them take JSONL for fine-tuning. It is not coincidence:

• Examples are independent. Training example 40,000 does not depend on example 1, which is exactly the model JSONL encodes.
• It appends. Datasets grow. Adding examples means writing more lines, not rewriting a document.
• Errors are locatable. "Line 8,412 has an empty assistant message" is actionable. "The JSON is invalid" is not.
• It streams. A 2 GB dataset needs no special handling on either side.
• It shards cleanly. Any newline is a safe split point, so parallel preprocessing is trivial.

The pipeline, in order

Order matters here more than people expect. Doing these steps in the wrong sequence is how leaked evaluation sets happen.

1. Get the format right. Convert whatever you have into the target provider's shape with the chat format converter — ShareGPT, Alpaca, OpenAI chat, Anthropic messages and Gemini all interconvert.
2. Validate syntax. The validator then the auto-fixer. Everything downstream assumes the file parses.
3. Clean the encoding. Strip BOMs, normalise line endings, remove invisible characters. Zero-width characters make identical-looking records compare as different, which quietly defeats deduplication.
4. Deduplicate. Before splitting, not after. This is the single most important ordering rule on this page — see below.
5. Remove PII. The anonymizer detects emails, phones, IPs, card and social-security numbers. A model trained on personal data can emit it verbatim to any user.
6. Check the token budget. Token counter tells you both the cost and which examples exceed the context window — those get silently truncated, teaching the model to stop mid-sentence.
7. Split. Train / val / test, or k-fold when data is scarce.
8. Verify no leakage. Check the eval set against training before you trust any number the run produces.
9. Validate against the provider's rules. OpenAI, Anthropic, Gemini, Llama, Mistral. Each has its own rules about role alternation, system prompts and empty content.

Deduplicate before you split

If the same example appears twice and you split afterwards, one copy can land in training and the other in evaluation. Your eval score then partly measures memorisation, and it will look better than the truth — which is the worst kind of wrong, because nothing prompts you to investigate a good number.

Deduplicate first, then split, then verify with the leakage detector. The verification step is cheap and catches the cases deduplication misses — near-duplicates, the same question with a reworded answer, and records duplicated across source files.

The related trap: related records must stay together. Several turns of one conversation, or several examples derived from one document, have to land in the same split. Otherwise the model sees part of a test item during training. Both splitters support grouping for exactly this.

Provider format differences that actually bite

The converter handles all of these. The one that catches people repeatedly is the Anthropic system prompt: moving it into the array produces a file that looks fine and is rejected.

Fine-tuning files are not batch files

Both are JSONL, both go to the same provider, and they are completely different formats. A fine-tuning line is a training example — {"messages": [...]}. A batch line is an API request, wrapping a full request body in custom_id / method / url / body.

Validate them with different tools: fine-tune validator versus batch validator. The quick tell is the first key — {"messages" means training, {"custom_id" means batch.

Preference data is a second dataset

DPO, ORPO and KTO train on pairs: for one prompt, a chosen and a rejected completion. The record shape differs from SFT, and so do the failure modes — an identical chosen and rejected pair teaches nothing, and a systematic length difference between them teaches verbosity rather than quality.

The DPO validator checks the structural problems and points at the content ones. Preference records are roughly twice the length of an SFT example, so re-check the token budget — context limits bite sooner than people expect.

Before you spend money

A short pre-flight list. Each of these has cost somebody a training run:

• Provider validator says clean, including warnings.
• Token count is inside the context window for every example, not on average.
• Leakage detector reports zero overlap between splits.
• Anonymizer has been run, and you have looked at the output rather than trusting it.
• A dozen random records read correctly by eye — the dataset editor is for exactly this. Automated checks cannot tell you the labels are right.
• Class balance is what you intended. Field coverage shows distinct-value counts.

Frequently asked questions

Tools mentioned on this page

• Chat format converterBetween every provider's conversation shape.
• Leakage detectorThe check that saves you from a meaningless score.
• Token counterCost, and which examples exceed the context window.
• AnonymizerRemove PII before training on it.

— S., [email protected]