Merge JSONL Files
Many files into one. Three collection runs, a hand-written batch and last week's file all need to become one training set — without keeping the rows that appear in two of them, and without ending up with three homogeneous blocks glued together. Drop them all in, choose how they interleave, and dedupe across the whole set in one pass.
Merge
Which ordering to use
- Concatenate when the files are already in a meaningful order, or when you want to be able to point at a range of lines and say which file it came from. This is the safe default and the arrows next to the file list control it.
- Interleave when the files are different kinds of example — one per task, one per source, one per annotator — and whatever reads the merged file might not shuffle. Round-robin gives an even mix at every point in the file, so a truncated read still sees all the sources.
- Shuffle when you want a genuinely random order and reproducibility. The seed is a string, and the same seed always produces the same order, which is what makes a shuffled dataset a thing you can rebuild rather than a thing you have to keep.
Deduplicating across files
Overlap between collection runs is the normal case, not the exception, and duplicated training examples are worth removing: they raise the effective weight of whatever they contain, and if the same row lands in both your training and validation split, your evaluation is measuring memorisation.
Whole record compares canonical JSON — keys sorted, so {"a":1,"b":2} and
{"b":2,"a":1} are recognised as the same record even though the raw lines are not equal.
That is the honest comparison for records that came out of different tools.
One key field compares only that field, which is what you want when the same example
was re-exported with a new timestamp or a different metadata block. Dotted paths reach into nested
objects — messages.0.content dedupes chat records on their first user message. A record
that does not have the field at all is never treated as a duplicate, because there is nothing to
compare; that is deliberate, so a partially-annotated file cannot collapse to one row.
Deduplication runs after ordering, so the survivor is the first occurrence in the merged order. If you want the copy from a particular file to win, move that file to the top of the list.
Stamping the source
Adding a _source field costs a few bytes per row and buys you the ability to answer
"where did this example come from" three months later — which is the question you will actually have.
It also lets you check the mix downstream: value counts on
_source tells you what proportion of the merged file each input contributed. Rename the
field if _source collides with something you already use; a record that is not an object
has nowhere to put it and is passed through unchanged.