JSONL on the Command Line

Language guide · updated 28 July 2026 · overview · spec · format comparison · examples · best practices · troubleshooting · FAQ · glossary

JSONL is the one JSON dialect that behaves properly with Unix tools: one record per line means wc, head, split, shuf, sort and grep all work on records without understanding JSON at all. Add jq for the structure-aware parts and you rarely need a script.

The essentials

# How many records?
wc -l < data.jsonl

# First and last few records, pretty-printed
head -n 3 data.jsonl | jq .
tail -n 3 data.jsonl | jq .

# Is every line valid JSON? (prints nothing if all good)
jq -e . data.jsonl > /dev/null || echo "invalid JSON somewhere"

# Which line is broken?
awk '{ print NR": "$0 }' data.jsonl | while IFS= read -r l; do
  n=${l%%:*}; rest=${l#*: }
  echo "$rest" | jq -e . >/dev/null 2>&1 || echo "line $n is invalid"
done

That last loop is slow on big files. For a fast, complete report with positions and suggested fixes, paste the file into the JSONL validator instead — it is a browser page, so nothing is uploaded.

jq: the parts you will actually use

jq -c (compact) is the flag that keeps output as JSONL rather than pretty-printed JSON. Almost every recipe below needs it:

# Pick fields
jq -c '{id, name, email}' data.jsonl > slim.jsonl

# Filter records
jq -c 'select(.status == "active")' data.jsonl > active.jsonl
jq -c 'select(.score > 0.8 and .lang == "en")' data.jsonl

# Records where a field is missing or null
jq -c 'select(.email == null)' data.jsonl

# Rename a key, add a computed one
jq -c '{user_id: .id, name, initial: (.name[0:1])}' data.jsonl

# Flatten one level of nesting
jq -c '. + .meta | del(.meta)' data.jsonl

# Extract a column as plain text (-r drops the quotes)
jq -r '.email' data.jsonl > emails.txt

# Count by a field
jq -r '.country' data.jsonl | sort | uniq -c | sort -rn

# JSONL to CSV
jq -r '[.id, .name, .email] | @csv' data.jsonl > out.csv

# JSON array to JSONL, and back
jq -c '.[]' array.json > data.jsonl
jq -s '.' data.jsonl > array.json

Those last two are worth memorising: .[] unwraps an array into a stream, and -s (slurp) collects a stream into an array. They are the entire JSON ↔ JSONL conversion in two characters each.

Splitting, sampling, shuffling

# Split into 100k-record chunks, named part-aa.jsonl, part-ab.jsonl, ...
split -l 100000 --additional-suffix=.jsonl data.jsonl part-

# Random sample of 1000 records
shuf -n 1000 data.jsonl > sample.jsonl

# Shuffle the whole file (needs to fit in memory)
shuf data.jsonl > shuffled.jsonl

# Deterministic shuffle — same order every run
shuf --random-source=<(yes 42) data.jsonl > shuffled.jsonl

# 80/20 train/test split after shuffling
shuf data.jsonl > s.jsonl
n=$(wc -l < s.jsonl); k=$(( n * 80 / 100 ))
head -n "$k" s.jsonl > train.jsonl
tail -n +"$((k+1))" s.jsonl > test.jsonl

--random-source=<(yes 42) is the GNU trick for a reproducible shuffle; macOS shuf (from coreutils) supports it too, but the BSD tools do not. For reproducibility that works everywhere, our seeded shuffle and train / val / test split take an explicit seed.

Deduplicating and joining

# Remove byte-identical lines, preserving order
awk '!seen[$0]++' data.jsonl > deduped.jsonl

# Deduplicate by a key field, keeping the first occurrence
jq -c '. as $r | .id' data.jsonl | \
  awk '!seen[$0]++ { print NR }' > keep.txt
awk 'NR==FNR { k[$1]; next } FNR in k' keep.txt data.jsonl > deduped.jsonl

# Join two files on id (both must be sorted by the key)
jq -r '[.id, tostring] | @tsv' a.jsonl | sort > a.tsv
jq -r '[.id, tostring] | @tsv' b.jsonl | sort > b.tsv
join -t$'\t' a.tsv b.tsv | cut -f2,3 > joined.tsv

The join recipe is where shell scripting stops being pleasant — the sorting, the TSV round-trip, the escaping. Our join tool does it on a key with inner, left and full semantics, and the SQL playground handles anything more complex with real SQL over DuckDB. awk '!seen[$0]++', by contrast, is genuinely the best way to drop identical lines — though note it compares bytes, so two records differing only in key order both survive. Our deduplicator compares canonically.

Gzipped files without unpacking

# Everything above works on .gz with a z-prefixed tool or a pipe
zcat data.jsonl.gz | wc -l
zcat data.jsonl.gz | jq -c 'select(.status == "active")' | gzip > active.jsonl.gz

# Peek without decompressing the whole thing
zcat data.jsonl.gz | head -n 3 | jq .

# Compress with maximum effort
gzip -9 data.jsonl        # replaces the file
gzip -9 -k data.jsonl     # keeps the original

On macOS use gzcat rather than zcat. Piping into head gives a SIGPIPE once head exits, which is harmless but prints a warning in some shells — 2>/dev/null silences it.

Watch out for these

• jq without -c destroys the format. Pretty-printed output spreads each record over many lines, so the result is no longer JSONL. This is the most common shell mistake with the format.
• grep on JSONL is a substring match, not a field match. grep '"status":"active"' works only if the writer used exactly that spacing. Use jq 'select(...)' for anything you care about.
• sort sorts lexically. Sorting JSONL text does not sort records by any field — it sorts by whatever the line starts with. Use the sorter or jq -s 'sort_by(.field)[]'.
• A missing final newline makes wc -l under-count by one and makes cat a.jsonl b.jsonl join two records. Check with tail -c1 file | xxd.
• split without -l splits by bytes and will cut a record in half.

Frequently asked questions

Tools mentioned on this page

• Query (jq-style)The same filters, in the browser, with instant feedback.
• SQL playgroundDuckDB over your file — much faster than jq for aggregation.
• Join two filesWithout the TSV round-trip.
• Shuffle (seeded)Reproducible on any platform.

— S., [email protected]