JSONL Troubleshooting
Most JSONL failures come from a handful of causes, and the error messages rarely name them. This page maps the message you actually see — Extra data, Unexpected token, Expecting value — onto what is really wrong and what to do about it.
"Extra data" / "Unexpected non-whitespace character after JSON"
What it means: something is parsing your JSONL file as a single JSON document. It read line 1 successfully, then found more content and complained.
Where you see it: Python's json.load(f), JavaScript's
JSON.parse(text), jq without -c on multi-value input, any
"upload a JSON file" form.
The fix: parse line by line, not all at once. json.loads per line in
Python, or convert the file to a JSON array first with
JSONL → JSON if the consumer genuinely needs one document. This is not a
corrupt file — it is the right file being read the wrong way.
"Expecting value: line 1 column 1"
Almost always a byte-order mark. An invisible U+FEFF sits before your opening brace, and the parser sees an unexpected character at the very first position. Editors do not show it, which is why the error looks impossible.
Confirm it in a shell:
head -c 3 data.jsonl | xxd
# efbbbf ← that is a UTF-8 BOM
The fix: strip it. Our encoding fixer removes it
along with other invisible characters, or in Python open the file with
encoding="utf-8-sig", which consumes a BOM if present and is harmless if not.
The other cause of this message is an empty or blank first line — some writers emit a leading newline. Skipping blank lines when you read is good practice regardless.
"Unexpected token" part-way through a line
Four usual suspects, in rough order of frequency:
- A trailing comma —
{"a": 1,}. Valid in JavaScript source, invalid in JSON. Common when records were assembled by string concatenation. - Single quotes —
{'a': 1}. That is a Python dict repr or a JavaScript literal, not JSON. If a Python script wrote it withstr(dict)instead ofjson.dumps, this is your problem, and note that Python'sTrue/False/Nonewill be wrong too. - Curly quotes —
{“a”: 1}. The file went through a word processor, chat client, or ticket comment. Visually nearly identical to straight quotes. - An unescaped newline inside a string — a literal line break in a value. This one also splits your record across two lines, so it breaks the JSONL contract as well as the JSON.
The fix: the Auto-fixer repairs all four and shows what it changed line by line. Run the encoding fixer first if curly quotes or invisible characters are involved — syntax errors are much easier to read once the invisible characters are gone.
The last line is broken and the rest is fine
The file is truncated. A process was killed mid-write, a disk filled up, a download was interrupted, or a container was stopped while flushing. The last record is a half-written fragment.
This is the failure mode JSONL handles best: every complete line before the break is still perfectly good. Drop the last line and carry on:
# Keep everything except the final line
sed '$d' data.jsonl > fixed.jsonlCompare against a JSON array, where a truncated file has no closing bracket and is entirely unparseable. That resilience is a large part of why log pipelines use JSONL.
"Token too long" / the reader stops early
A line-length limit. The classic case is Go's bufio.Scanner, which gives
up on lines over 64 KB — and if you do not check sc.Err(), it stops silently and
your program exits successfully having processed half the file.
Java's BufferedReader, some Node stream configurations, and several log shippers have
similar caps. The tell is a record count that is lower than wc -l reports, with no error.
The fix: raise the limit (see JSONL in Go for the exact call), and always check the scanner's error. If records are genuinely enormous, truncate the offending fields or move large blobs out of the record and reference them.
Numbers changed value
Floating-point precision. JavaScript and JSON parsers in many languages decode all
numbers as IEEE 754 doubles, which represent integers exactly only up to 253. An id like
9007199254740993 comes back as 9007199254740992. Nothing warns you; the file
was fine and the parse silently changed the data.
The fix: keep large identifiers as strings in the JSONL. If you cannot change
the producer, parse with something that preserves the literal — json.Number in Go,
parse_int=str in Python's json.loads, a BigInt-aware library in JavaScript.
Snowflake ids, Twitter ids and some database sequences all exceed the safe range.
Non-English text turned into question marks or �
An encoding mismatch that already happened. The bytes were decoded with the wrong
character set — usually Windows-1252 bytes read as UTF-8, or the reverse. Once a character has become
� (U+FFFD), the original is gone: the replacement character carries no information
about what it replaced.
The fix is upstream. Re-export as UTF-8, or if you still have the original bytes,
convert them properly: iconv -f windows-1252 -t utf-8 in.jsonl > out.jsonl. Our
encoding fixer counts replacement characters so you know to go back
rather than trying to patch the symptom.
A separate, benign case: \u00e9 style escapes are valid JSON and parse back to
é correctly. Ugly in a diff, harmless in data. In Python they come from
ensure_ascii=True, the default.
Records look right but a field is always empty
Three quiet causes, none of which raise an error:
- A typo in a struct tag or model field. Go and many deserialisers ignore unknown fields by default, so a misspelled tag means the field is silently always zero. Turn on strict decoding while developing.
- Inconsistent key naming between producers.
user_idin most records anduserIdin some. Field coverage makes this obvious — you will see two near-identical field names with complementary percentages. - The field is nested deeper than you think.
meta.user.idrather thanuser_id. The schema inferrer shows the real structure.
Frequently asked questions
What is the fastest way to find which line is bad?
Paste the file into the JSONL validator — it reports every bad line with position, message and a suggested fix, in one pass, without uploading anything. Shell loops that pipe each line through jq work but are slow.
Can a JSONL file have blank lines?
The format says one JSON value per line, so strictly no. In practice most readers skip them and most writers occasionally emit them. Skip them when reading, and strip them with the encoding fixer if a strict consumer objects.
Does JSONL need a trailing newline at the end?
It should have one. Without it, concatenating two files joins the last record of the first to the first record of the second, and wc -l under-counts. Every tool here writes one.
Why does my file work in Python but not in Node?
Usually a BOM (Python's utf-8-sig handles it, JSON.parse does not) or a large integer (Python keeps arbitrary precision, JavaScript does not). Both are covered above.
Is a 1 GB JSONL file too big?
Not inherently — streaming readers use constant memory. It is too big for anything that loads the whole file, which includes most "paste your JSON here" websites. The tools here stream, so 1 GB works in the browser.
Tools mentioned on this page
— S., [email protected]