JSONL in Go

Language guide · updated 28 July 2026 · overview · spec · format comparison · examples · best practices · troubleshooting · FAQ · glossary

Go's encoding/json handles JSONL two different ways, and picking the wrong one is the usual source of trouble. json.Decoder streams whitespace-separated values for free; bufio.Scanner gives you real line control but has a token-size limit that bites on long records.

json.Decoder — the shortest correct version

A json.Decoder reads successive JSON values from a stream, and JSONL is exactly that. No line splitting needed:

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "os"
)

type Record struct {
    ID   int    `json:"id"`
    Name string `json:"name"`
}

func main() {
    f, err := os.Open("data.jsonl")
    if err != nil {
        panic(err)
    }
    defer f.Close()

    dec := json.NewDecoder(f)
    for {
        var rec Record
        if err := dec.Decode(&rec); err == io.EOF {
            break
        } else if err != nil {
            panic(err)
        }
        fmt.Println(rec.ID, rec.Name)
    }
}

This is idiomatic and streams properly. Its one limitation: because it does not care about lines, it cannot tell you which line failed. dec.InputOffset() gives a byte offset, which you can turn into a line number, but if error reporting per line matters, use the scanner instead.

bufio.Scanner — when you need line numbers

func readJSONL(path string) error {
    f, err := os.Open(path)
    if err != nil {
        return err
    }
    defer f.Close()

    sc := bufio.NewScanner(f)
    // The default token limit is 64 KB — far too small for real records.
    sc.Buffer(make([]byte, 0, 1024*1024), 16*1024*1024) // up to 16 MB per line

    for lineno := 1; sc.Scan(); lineno++ {
        line := bytes.TrimSpace(sc.Bytes())
        if len(line) == 0 {
            continue
        }
        var rec Record
        if err := json.Unmarshal(line, &rec); err != nil {
            return fmt.Errorf("line %d: %w", lineno, err)
        }
        // use rec
    }
    return sc.Err()
}

That sc.Buffer call is the important line. Without it, bufio.Scanner stops with bufio.ErrTooLong on any line over 64 KB — and it stops quietly if you forget to check sc.Err(), so your program processes half the file and exits successfully. It is the single most common JSONL bug in Go.

Writing JSONL

json.Encoder already appends a newline after each value, so encoding in a loop produces valid JSONL with no manual concatenation:

func writeJSONL(path string, recs []Record) error {
    f, err := os.Create(path)
    if err != nil {
        return err
    }
    defer f.Close()

    w := bufio.NewWriter(f)
    defer w.Flush()

    enc := json.NewEncoder(w)
    enc.SetEscapeHTML(false)   // keep < > & literal instead of \u003c
    for _, rec := range recs {
        if err := enc.Encode(rec); err != nil {   // Encode adds the "\n"
            return err
        }
    }
    return nil
}

SetEscapeHTML(false) is worth knowing about: Go escapes <, > and & by default, which is safe for embedding JSON in HTML and just noise in a data file. Never call SetIndent — indentation would spread one record across several lines and break the format.

Mixed shapes with json.RawMessage

When records do not share one struct, decode the part you know and defer the rest. This is also how you handle a discriminator field:

type Envelope struct {
    Type    string          `json:"type"`
    Payload json.RawMessage `json:"payload"`
}

var env Envelope
if err := json.Unmarshal(line, &env); err != nil {
    return err
}
switch env.Type {
case "user":
    var u User
    if err := json.Unmarshal(env.Payload, &u); err != nil {
        return err
    }
case "order":
    var o Order
    if err := json.Unmarshal(env.Payload, &o); err != nil {
        return err
    }
}

json.RawMessage holds the bytes without parsing them, so this costs one pass rather than two. To find out what shapes are actually in a file before writing the structs, our field coverage report lists every key with the types observed, and the schema inferrer produces a JSON Schema you can generate Go types from.

Gzip and concurrency

// Read .jsonl.gz
f, _ := os.Open("data.jsonl.gz")
defer f.Close()
gz, err := gzip.NewReader(f)
if err != nil {
    panic(err)
}
defer gz.Close()

sc := bufio.NewScanner(gz)
sc.Buffer(make([]byte, 0, 1024*1024), 16*1024*1024)
for sc.Scan() {
    // ...
}

Parsing is often the bottleneck, and JSONL parallelises trivially because every line is independent. Read lines in one goroutine, fan them out to a pool of workers over a channel, and fan results back in. Do not try to parallelise the reading itself — the disk or the decompressor is sequential, and the win is all in the unmarshalling.

Go-specific gotchas

• The 64 KB scanner limit, again, because it is the one that costs people hours. Always call sc.Buffer and always check sc.Err().
• Unknown fields are silently ignored. A typo in a struct tag means the field is quietly always zero. dec.DisallowUnknownFields() turns that into an error while you are developing.
• Missing and zero are indistinguishable for value types — an absent count and "count": 0 both give you 0. Use a pointer (*int) when you need to tell them apart.
• Numbers decode to float64 in an interface{}, so a large integer id loses precision. Use dec.UseNumber() to get a json.Number that preserves the literal text.

Frequently asked questions

Tools mentioned on this page

• Field coverageSee every key and its types before writing structs.
• Schema inferrerGenerate a JSON Schema to base Go types on.
• JSONL ValidatorConfirm the file is really one value per line.
• TruncateMake a small fixture for your tests.

— S., [email protected]