JSONL Glossary

Reference · updated 28 July 2026 · overview · spec · format comparison · examples · best practices · troubleshooting · FAQ

The vocabulary that shows up in JSONL documentation, error messages and data-engineering conversations — defined plainly, with links to the tool or guide where each idea becomes practical.

Format terms

JSONL / JSON Lines: A text format where each line is one complete JSON value. No enclosing array, no separating commas. Overview.
NDJSON: Newline-Delimited JSON. The same format under a different name, from a separate specification effort. Comparison.
Record: One line, one JSON value. The unit everything operates on: you count records, filter records, split on record boundaries.
Line-delimited: The property that a newline separates units. It is what makes split, head and shuf work on JSONL without them understanding JSON.
Heterogeneous records: Records in one file with different shapes — normal for event logs, awkward for typed consumers. Field coverage shows how uniform a file really is.
Envelope: A wrapper object carrying metadata plus a payload, e.g. {"type": "user", "ts": …, "payload": {…}}. Lets one stream carry several record types with a discriminator.
Discriminator: The field that says which shape a record has — usually type or event. Read it first, then parse the payload accordingly.

Processing terms

Streaming: Processing records as they arrive rather than loading everything first. Memory stays constant regardless of file size — the main reason JSONL exists.
Backpressure: A slow consumer signalling a fast producer to wait. Ignoring it in Node (writing without awaiting drain) buffers the whole file in memory, defeating the point of streaming. Example.
Splittable: Whether a file can be divided for parallel processing without parsing it. JSONL is: any newline is a safe boundary. A JSON array is not. This is why Spark and Hadoop like the format.
Sharding: Splitting one logical dataset across many files, usually so they can be processed in parallel or stay under a size limit. Splitter.
Checkpointing: Recording how far a job got — for JSONL, usually a line number or byte offset — so a restart resumes instead of redoing everything.
Idempotent ingestion: Loading the same records twice without creating duplicates, typically by keying on a stable id. Relevant whenever a retry might re-send a shard.
Flattening: Turning nested structure into dotted keys — user.address.city — so the data fits a table. Required for CSV and most SQL targets. Flatten / unflatten.
Canonical JSON: A single byte-exact representation of a document: keys sorted, no insignificant whitespace, numbers in shortest form. Needed for hashing, signing and reliable deduplication, and standardised as RFC 8785. Deduplicator uses it.

Data-quality terms

Schema drift: The shape of records changing over time — a field added, renamed, or its type changed — without the consumers being told. The usual cause of a pipeline that worked yesterday. Schema validator.
Schema inference: Deriving a schema by observing data rather than being given one. Only as good as the sample: a field absent from every sampled record looks like it does not exist. Schema inferrer.
Coverage: The proportion of records in which a field is present at all. A field at 3% coverage is usually either rare or a misspelling of one at 97%. Field coverage.
Cardinality: How many distinct values a field holds. Two distinct values across 50,000 records means it is a flag; one per record means it is an identifier.
Deduplication: Removing repeated records — either byte-identical, canonically identical, or sharing a key. Which definition you use changes the answer. Deduplicator.
PII: Personally identifiable information: names, emails, phone numbers, addresses, IPs. Needs removing or masking before a dataset is shared or used for training. Anonymizer.
Truncation: A file cut off mid-write, leaving a partial final line. JSONL degrades gracefully here — every complete line before the break is still valid. Recovery.

Machine-learning terms

SFT — supervised fine-tuning: Training on input/output pairs — the conversations in a normal fine-tuning JSONL file. The first stage of most fine-tuning pipelines.
DPO — direct preference optimisation: Training on preference pairs: for one prompt, a chosen and a rejected completion. The model learns from the difference. DPO validator.
Train / validation / test split: Partitioning data so the model is trained on one part, tuned on a second, and evaluated once on a third. Splitter.
K-fold cross-validation: Splitting into k folds and training k times, each with a different fold held out. Lower-variance estimates at k times the cost. K-fold split.
Data leakage / contamination: Evaluation examples that also appear in training. Produces scores that look excellent and mean nothing. Leakage detector.
Stratification: Splitting so each part keeps the same class proportions as the whole. Matters most when a class is rare enough to land entirely in one fold by chance.
Token: The unit a model actually consumes — roughly a word-piece. Context limits and pricing are both counted in tokens, not characters. Token counter.
ShareGPT / Alpaca format: Two community conventions for conversation datasets: ShareGPT uses conversations with from/value, Alpaca uses instruction/input/output. Converter.
Batch API: Submitting many requests as one JSONL file for cheaper asynchronous processing. A different format from fine-tuning files, and easy to confuse. Batch validator.

Frequently asked questions

Is there an official JSONL glossary?

No. The format has no standards body, so terminology comes from the tools and the surrounding data-engineering practice. This page documents the usage you will actually meet.

What is the difference between a record and a line?

In valid JSONL, nothing — one line is one record, which is the whole contract. They diverge only in broken files: a truncated write or an unescaped newline can leave a record spread across two lines.

Tools mentioned on this page

— S., [email protected]