JSONL in Python

Language guide · updated 28 July 2026 · overview · spec · format comparison · examples · best practices · troubleshooting · FAQ · glossary

Python has no jsonl module and does not need one — a JSONL file is just lines of JSON, so a three-line loop handles it. This page covers the plain-stdlib approach, pandas and Polars for dataframes, orjson when parsing is the bottleneck, and how to stream a file far larger than memory.

The stdlib way — no dependencies

This is the whole technique. json.loads on each line, and skip blank lines:

import json

def read_jsonl(path):
    with open(path, encoding="utf-8") as fh:
        for line in fh:
            line = line.strip()
            if line:                      # tolerate blank lines
                yield json.loads(line)

for record in read_jsonl("data.jsonl"):
    print(record["id"])

Writing is the mirror image. The two details that matter are both easy to get wrong:

import json

def write_jsonl(path, records):
    with open(path, "w", encoding="utf-8") as fh:
        for record in records:
            # ensure_ascii=False keeps UTF-8 readable instead of \uXXXX escapes.
            # No indent — indent would put a record across several lines and
            # break the one-record-per-line contract.
            fh.write(json.dumps(record, ensure_ascii=False) + "\n")

Because it is a generator, memory use stays flat no matter how big the file is — one record at a time. That is the property that makes JSONL worth using in the first place.

Skipping bad lines instead of crashing

One malformed line should not kill a job that has already processed two million good ones. Report the line number and move on:

import json

def read_jsonl_tolerant(path):
    errors = []
    with open(path, encoding="utf-8") as fh:
        for lineno, line in enumerate(fh, start=1):
            line = line.strip()
            if not line:
                continue
            try:
                yield json.loads(line)
            except json.JSONDecodeError as exc:
                errors.append((lineno, str(exc)))
    if errors:
        print(f"{len(errors)} bad line(s); first: line {errors[0][0]} — {errors[0][1]}")

If you would rather see the whole error list before writing any code, paste the file into the JSONL validator — it reports every bad line with its position and a suggested fix, and the Auto-fixer repairs the common cases.

pandas: read_json with lines=True

For tabular JSONL, pandas reads it in one call. The lines=True argument is the whole trick:

import pandas as pd

df = pd.read_json("data.jsonl", lines=True)

# Write it back out
df.to_json("out.jsonl", orient="records", lines=True, force_ascii=False)

For a file that will not fit in memory, chunksize gives you an iterator of dataframes:

reader = pd.read_json("big.jsonl", lines=True, chunksize=50_000)
total = 0
for chunk in reader:
    total += chunk["amount"].sum()

Two pandas-specific traps. First, it infers dtypes per chunk, so a column that is integer in chunk 1 and float in chunk 2 gives you inconsistent types — pass dtype= explicitly for anything load-bearing. Second, it parses date-looking strings into timestamps by default; convert_dates=False turns that off when you want the raw strings.

Polars: usually the faster choice

Polars reads NDJSON natively and is typically several times faster than pandas on the same file, with lower memory use:

import polars as pl

df = pl.read_ndjson("data.jsonl")

# Lazy + streaming: never materialises the whole file
result = (
    pl.scan_ndjson("big.jsonl")
      .filter(pl.col("status") == "active")
      .group_by("country")
      .agg(pl.col("amount").sum())
      .collect(streaming=True)
)

scan_ndjson with collect(streaming=True) is the closest thing Python has to running SQL over a JSONL file without a database. If you would rather write actual SQL, our SQL playground does it in the browser with DuckDB — and DuckDB's Python package does the same locally with read_json_auto('data.jsonl').

orjson when parsing is the bottleneck

If profiling says json.loads dominates, orjson is a drop-in replacement that is typically 2–5× faster. It takes and returns bytes, so skip the decode step entirely:

import orjson

with open("data.jsonl", "rb") as fh:      # binary mode — no decode per line
    for line in fh:
        if line.strip():
            record = orjson.loads(line)

# Writing: orjson.dumps returns bytes and never adds whitespace
with open("out.jsonl", "wb") as fh:
    for record in records:
        fh.write(orjson.dumps(record) + b"\n")

Note orjson.dumps has no ensure_ascii option — it always emits UTF-8, which is what you want for JSONL anyway. It also serialises datetime, UUID and numpy types that the stdlib refuses without a custom encoder.

Gzipped JSONL

.jsonl.gz is extremely common, because repeated keys compress by 80–90%. Python reads it with a one-word change — gzip.open instead of open:

import gzip, json

with gzip.open("data.jsonl.gz", "rt", encoding="utf-8") as fh:
    for line in fh:
        if line.strip():
            record = json.loads(line)

The "rt" mode matters: "rb" gives you bytes, which json.loads accepts but line.strip() then compares against a str. To see how much a particular file gains from compression before committing, our gzip tool reports the exact ratio in the browser.

Which approach to reach for

Frequently asked questions

Tools mentioned on this page

• JSONL ValidatorFind every bad line before writing parsing code.
• Auto-fixerRepair the broken lines the validator reports.
• SQL playgroundQuery a JSONL file with DuckDB, no Python needed.
• Flatten / unflattenTurn nested records into flat columns for pandas.

— S., [email protected]