JSONL for Structured Logging

Use case · updated 28 July 2026 · overview · spec · format comparison · examples · best practices · troubleshooting · FAQ · glossary

Structured logging means each log line is a JSON object rather than a formatted sentence — which is to say, your log file is a JSONL file. That one change turns grepping for patterns into querying fields, and makes a log pipeline something you can reason about.

Why the format fits logging so exactly

Append-only by nature. A log is a sequence of independent events. Writing one more line requires no rewrite — the same property that makes JSONL cheap makes logging cheap.
Crash-tolerant. If the process dies mid-write, only the final partial line is lost. Every earlier line is still valid. A JSON array log would be entirely unparseable.
Line-oriented tools still work. tail -f, grep, wc -l, log rotation, split — none of them need to understand JSON.
Every shipper speaks it. Fluent Bit, Vector, Logstash, Filebeat, Promtail, and every cloud logging agent read newline-delimited JSON natively.
It compresses hard. Repeated keys across millions of lines gzip to a fraction of the original — often 90% for logs specifically.

Fields worth standardising

The value of structured logs comes from consistency, and consistency comes from agreeing on field names before there are ten services. A workable baseline:

{"ts":"2026-07-28T10:14:22.481Z","level":"error","msg":"payment declined","service":"checkout","env":"prod","trace_id":"4bf92f...","span_id":"00f067...","user_id":"u_8812","duration_ms":142,"err":{"type":"CardDeclined","code":"insufficient_funds"}}

Field	Why
`ts`	ISO 8601 with milliseconds and an explicit `Z`. Never a local time without an offset.
`level`	A small fixed set — `debug`, `info`, `warn`, `error`, `fatal`. Lowercase, consistently.
`msg`	A short, constant string. Put the variable parts in their own fields — see below.
`service`, `env`	So one aggregated stream can be filtered back apart.
`trace_id`, `span_id`	The link between a log line and a distributed trace. Worth adding before you need it.
`duration_ms`	Numeric, one unit, named with the unit. `duration` alone invites ambiguity.
`err`	A nested object with a stable `type` and `code`, not a stringified stack trace.

Keep the message constant, the data in fields

The mistake that undermines structured logging is interpolating values into the message:

// Hard to aggregate — every line is a unique string
{"level":"error","msg":"payment for user u_8812 declined: insufficient_funds"}

// Groupable — the message is a stable key
{"level":"error","msg":"payment declined","user_id":"u_8812","reason":"insufficient_funds"}

With a constant msg, "how many payment declines yesterday, grouped by reason" is one query. With interpolation it is a regex you will get wrong. The rule: the message names the event, fields carry the specifics.

The related discipline is bounded cardinality in fields you group by. A user_id field is fine as data; using it as a metric label creates a series per user and will melt your monitoring bill.

Analysing a log file you already have

# Errors in the last file, newest first
jq -c 'select(.level == "error")' app.jsonl | tail -20 | jq .

# Count by level
jq -r '.level' app.jsonl | sort | uniq -c | sort -rn

# Slowest requests
jq -c 'select(.duration_ms > 1000) | {ts, msg, duration_ms}' app.jsonl

# Error types, ranked
jq -r 'select(.level=="error") | .err.type // "unknown"' app.jsonl | sort | uniq -c | sort -rn

# Everything for one trace, across services
zcat *.jsonl.gz | jq -c 'select(.trace_id == "4bf92f...")'

Once questions involve grouping and time buckets, jq stops being the right tool. DuckDB reads JSONL directly and is dramatically faster:

SELECT date_trunc('hour', CAST(ts AS TIMESTAMP)) AS hour,
       level,
       count(*) AS n,
       quantile_cont(duration_ms, 0.95) AS p95
FROM read_json_auto('app*.jsonl.gz')
GROUP BY 1, 2
ORDER BY 1 DESC;

Our SQL playground runs the same engine in the browser, which is a convenient way to interrogate a log file someone sent you without setting anything up — and without the file leaving your machine.

Rotation, compression, retention

Rotate by size or time, and always at a line boundary. A rotation that cuts mid-line breaks two records rather than one.
Compress rotated files to .jsonl.gz. Logs are the best case for gzip, and every analysis tool above reads gzip directly, so nothing gets harder.
Name files sortably — app-2026-07-28.jsonl.gz — so a glob processes them in order.
Sample the noisy levels rather than dropping them. Keeping 1% of debug preserves the shape of a problem while cutting the volume; dropping it entirely removes the context you want during an incident.
Scrub PII on the way in, not out. Once a token or an email is in your log store it is in your backups. The anonymizer shows what a scrubbing pass catches; the real fix is a redacting formatter in the logger.

Before shipping a log file to anyone

Log files are, empirically, full of things that should not leave the building: session tokens in URLs, emails in error messages, internal hostnames, full request bodies.

Field coverage — see every field that exists, including the ones you forgot are logged.
Anonymizer — mask emails, IPs, card numbers and tokens.
Minifier with a remove-keys list — drop whole fields you cannot share.
Truncate — cut giant request bodies down to something reviewable.
Read a sample by eye. Every automated pass misses something.

Frequently asked questions

Is JSON logging slower than plain text?

Marginally, at the write. Serialising an object costs more than formatting a string, and the lines are longer. In exchange you get queryable fields, which pays for itself the first time you need to aggregate. Every mainstream logging library has a fast JSON encoder.

Should timestamps be ISO strings or epoch numbers?

ISO 8601 with an explicit offset. It is human-readable in tail -f, sorts lexically in the right order, and every tool parses it. Epoch numbers are smaller and unreadable at 3am during an incident.

How do I handle multi-line stack traces?

Put the trace in a field with escaped newlines — "stack": "line1\nline2". That keeps one event on one line, which is the whole contract. Log shippers with multi-line reassembly rules exist precisely because unstructured logs break it.

What about very high volume?

Sample at the source, keep debug off in production, and compress on rotation. If a single service is writing enough JSON that serialisation shows up in a profile, that is worth measuring — but it is rarer than people assume.

Can I query gzipped logs without decompressing?

Yes. zcat file.jsonl.gz | jq streams it, and DuckDB reads .gz directly in read_json_auto. No temporary files needed.

Tools mentioned on this page

— S., [email protected]