Skip to content
FmtTools

Workflow toolkit

Tools for Reading Structured Logs

For the log line that is valid JSON on its own but invalid as a file: NDJSON, interleaved writes, embedded base64, and how to get a queryable array.

Updated
November 17, 2026
Reading time
17 min read
Length
3,700 words
Reviewer note
Checked against the published specification
A four-panel illustration of structured-log debugging: a newline-delimited file of JSON records, a torn line split between two writers, an escaped JSON body inside one string field, and a decoded base64 payload with its magic bytes.

A production log file is usually a very good JSON document and a very bad one at the same time: every line is valid, and the file as a whole is a syntax error. This page is the runbook for that gap. It starts with the grammar rule that causes the problem, then works through the things that make real logs hard to read after that: records that were written by two processes at once, JSON buried inside a string, base64 buried inside a field, timestamps in three different units, identifiers that lose digits when they pass through a number, and single lines large enough to defeat a clipboard.

Start here

The one-value rule and recognising NDJSON

RFC 8259 defines a JSON text in one production: JSON-text = ws value ws. One value, optionally surrounded by whitespace. There is no production for a second value, and no separator token, which means a file that contains {"level":"error"} followed by a line feed and then {"level":"info"} is not a JSON text at all. Everything else about newline-delimited logging is a convention layered on top of that silence. The JSON Lines convention states that each line is a valid JSON value, the terminator is a line feed, and empty lines are not allowed. The same convention is published under the name NDJSON, with the media type application/x-ndjson. None of that is an RFC.

Paste the four-record file below into the JSON validator and the report is deterministic. The scanner consumes the first object completely, skips the line feed, finds another value where the grammar allows none, and stops with the message More than one value at the top level. at line 2, column 1, offset 118.

four records, 434 bytes4 lines
{"ts":"2026-11-12T03:14:15.926Z","level":"error","svc":"checkout","msg":"payment gateway timeout","trace":"a3f19c2e"}
{"ts":"2026-11-12T03:14:15.927Z","level":"info","svc":"cart","msg":"cart rebuilt","user":"8741220"}
{"ts":"2026-11-12T03:14:16.004Z","level":"warn","svc":"checkout","msg":"retry 1 of 3","trace":"a3f19c2e"}
{"ts":"2026-11-12T03:14:16.411Z","level":"info","svc":"checkout","msg":"payment accepted","trace":"a3f19c2e"}

117 + 99 + 105 + 109 bytes of records, plus four line feeds, is 434 bytes. Nothing here is malformed; all four lines are valid JSON documents.

What each framing is, and where its rules come from
FramingRecord delimiterStandard statusWhat a JSON parser does with it
One JSON array of objectsCommas, brackets anywherePlain RFC 8259Parses. This is the target shape for the tools on this page.
NDJSON / JSON LinesLine feed (U+000A) after each valueConvention with its own spec site, not an RFCError at the second value: more than one value at the top level.
JSON text sequenceRecord separator U+001E before each valueRFC 7464, media type application/json-seqError at the RS byte: the control character cannot start a value.
Pretty-printed array in one fileWhitespace, significant only inside stringsPlain RFC 8259Parses, but is useless for line-based shipping and forbidden in bulk bodies.
Concatenated objects, no whitespaceNoneNot a format; called JSON streams informallyError at the second opening brace, and repair mode may guess.

Only two of those five are specified by an RFC: plain JSON and JSON text sequences. Everything else is convention.

Recognising the format before you parse it

Before you convert anything, confirm the diagnosis. The JSON formatter does this with a second, independent check: after the strict parse fails, it splits the input on line breaks, trims each line, discards the empty ones, and if at least two non-blank lines remain it validates every one on its own. Only when all of them parse does the error banner add a sentence: every line parses on its own, so this looks like newline-delimited JSON with N records. That number is a record count obtained without converting anything, and it is a strictness test: N records, all parsing individually, means the NDJSON hypothesis holds and the file is safe to wrap. If the sentence does not appear, the reason is almost always a torn line, a multi-line value, a header or footer that is not JSON, or a truncated final record.

Counting records four ways, and what each disagreement means
MethodCountsFails whenDisagreement tells you
Formatter noteNon-blank lines that each parse aloneany line is torn or multi-linea silent stop: the note is absent, so the file is not clean NDJSON
Input pane line statLine feeds plus onethe file has no trailing newline, or a value contains oneoff by one at the tail is normal; off by more is a multi-line record
Byte arithmeticSum of record bytes plus one LF eachrecords contain a line feed or the encoding is not UTF-8bytes above the sum means embedded newlines or multibyte characters
Stats after wrappingRecords in the array rootthe document fails to parsefewer records than lines means a lost record

Run all four on the same file before you analyse it. This is the whole of the integrity check.

Wrapping lines and JSON text sequences

Three ways get a file of records into a JSON array, and they are not equivalent. One preserves every digit of every number and every duplicate key; one of them does not, and the reason is where the bytes go.

The three conversion routes and what each one guarantees
RouteWhat happens to the textPreserves digitsReports record count
Wrap by hand, then strict formatOnly structural characters are added; record text is untouchedYes, numbers print from the parse treeCount it yourself, then check Records in the stats
Repair mode in the formatterThe stream is re-parsed into a value and re-serialisedNo: values pass through a doubleYes, in the fix list: N top-level values became 1 array
A streaming converter outside the tabDepends on the tool; most re-serialiseUsually noYes, and it works above tab memory limits

If the file is small enough to paste and contains no long numeric identifiers, the first route is the safest and the cheapest.

The transformation is three characters of punctuation per boundary. A bracket before the first line, a comma in place of each line feed between records, and a bracket after the last. For the four-record file: 430 bytes of records plus three commas plus two brackets is 435 bytes, one byte more than the 434-byte file it came from. Nothing else changes. Then format it. The JSON formatter prints from the parse tree rather than from a rebuilt value, which is the detail that matters: numeric literals are re-emitted exactly as written, so a 19-digit identifier survives as 19 digits even though no double could hold it. Key order survives, and duplicate keys are printed as both members instead of silently collapsing.

wrapped: 435 bytes, still one line per record4 lines
[
{"ts":"2026-11-12T03:14:15.926Z","level":"error","svc":"checkout","msg":"payment gateway timeout","trace":"a3f19c2e"},
{"ts":"2026-11-12T03:14:15.927Z","level":"info","svc":"cart","msg":"cart rebuilt","user":"8741220"}
]

Two of the four records shown for space. Mixing the comma and the line feed is legal: whitespace between values is insignificant.

Repair mode exists for JavaScript-flavoured payloads, and it also happens to collect a top-level sequence into an array. Its fix list is explicit: it found N values where RFC 8259 allows exactly one and collected them into one array. The mechanism, however, is re-serialisation: a number too large for a double is rounded before it is printed, and a key that appears twice keeps its last value. None of this makes repair mode wrong; it makes it a transform rather than a viewer, and a transform needs a verification step.

Records that contain newlines: JSON text sequences

NDJSON has one fatal assumption: the line feed cannot appear inside a record. RFC 7464 solves this by prefixing every element with RS, the record separator control character U+001E, and terminating with a line feed. The media type is application/json-seq. The consequence of putting the delimiter before the value rather than after it is the point of the whole design: a consumer that is cut off mid-element can tell, because the partial element has no following RS. Paste a real sequence into the JSON validator and you get an unexpected token at line 1, column 1: the leading RS byte is a control character that cannot start a value. The fix is mechanical: replace each RS with nothing, keep the line feeds, and treat the result as NDJSON.

Choosing between the two record framings for a file you control
PropertyNDJSON / JSON LinesJSON text sequence (RFC 7464)
StandardConvention with its own spec siteAn RFC, with a registered media type
DelimiterLine feed after each recordRecord separator U+001E before each record
A record may contain a raw line feedNo, and nothing detects the violationYes; the framing does not care about line feeds
Truncated final recordIndistinguishable from a recordDetectable: no closing delimiter after it
Readable with grep and jqYes, universallyRarely; jq does not read it without a preprocessing pass
Who uses itLog shippers, bulk APIs, Kafka file sinkMongoDB extended JSON, streaming protocols

Pick NDJSON when humans must inspect the file and the writer escapes strings correctly. Pick sequences when it may not, or when a stream can be cut mid-record.

Concurrency

Torn lines and JSON trapped inside strings

A torn line is what happens when two processes write to the same file and neither write is the unit of atomicity you assumed. The usual mechanisms: two descriptors opened without append mode, a buffered writer whose flush boundary falls inside a record, a record larger than the atomic-write limit of the channel it travels through, or a rotation that swaps the inode between a record's head and its tail. In every case the JSON is not corrupted. Two complete records are interleaved into one long byte string, and the fragments are what you have to recognise.

Take the fourth record of the sample file, the 109-byte line that ends payment accepted. Cut it at 77 bytes and the validator gives you a specific answer, because the cut falls inside a string: the first error is String is never closed. at line 1, column 72, offset 71. The remaining 32 bytes produce an equally specific answer: nt accepted","trace":"a3f19c2e" starts with a bare word, so the scanner reads it as an unquoted key.

one record, two fragments5 lines
HEAD (77 bytes)  {"ts":"2026-11-12T03:14:16.411Z","level":"info","svc":"checkout","msg":"payme
                 error: String is never closed.   line 1, col 72

TAIL (32 bytes)  nt accepted","trace":"a3f19c2e"}
                 error: "nt" is not quoted.       line 1, col 1

77 plus 32 is the original 109. The error pair, an unterminated string in one fragment and an unquoted key at column 1 in the other, is the signature of a split record.

The second pattern is the more dangerous one, because it imitates a different problem. When a writer finishes a record and another writer's leftover text follows the closing brace on the same line, the error is the same More than one value at the top level. you get from a whole NDJSON file. Read the offset: on a real NDJSON file it points at the start of line 2, and on an interleaved line it points at a position inside line 1. Same message, two entirely different causes, and the offset is the only thing that distinguishes them. The formatter's per-line check is a torn-line detector by construction: the newline-delimited note appears only when every non-blank line parses alone.

Fragments, what the validator reports, and what the fragment is worth
Fragment shapeReported errorRecoverable?What to do with it
Head cut inside a string valueString is never closed, at the value quoteYes, if the tail is foundConcatenate with the matching tail, then re-validate
Tail beginning with bare textAn unquoted key at column 1YesMatch it to a head by byte length; discard if the head is gone
Complete object plus orphan text on one lineMore than one value at the top levelYes: keep the object, drop the orphanCheck the offset against the line start before blaming NDJSON
Record missing its closing brace at EOFObject is missing its closing braceOnly by truncating the fileTail-follow with a framing-aware reader, or accept sequences
Two full records on one line, no fragmentsMore than one value at the top levelYes, triviallyA writer that omitted its line feed; fix the appender

Prevention belongs to the writer, not the reader: one write call per record, append mode, a per-process file when several processes share a directory.

JSON trapped inside a string field

The most common structural surprise in application logs is a field whose value is a JSON document that has been serialised into a string. It happens whenever a framework logs a request body by calling its JSON encoder twice. The result is legal, self-describing and unreadable. The inner document below is 117 bytes and contains twenty quote characters. Stored as a JSON string value it becomes 139 bytes: twenty backslashes plus the two quotes that wrap the value.

a stringified body inside a log record1 lines
{"ts":"2026-11-12T03:14:16.411Z","level":"debug","svc":"checkout","msg":"upstream request","body":"{\"method\":\"POST\",\"path\":\"/v2/payments\",\"headers\":{\"Content-Type\":\"application/json\"},\"amount\":49.99,\"currency\":\"EUR\"}"}

238 bytes. The body value is one JSON string of 139 bytes whose text is a 117-byte JSON document.

Un-nesting is two steps and one paste. Take the characters between the outer quotes, leaving the outer quotes behind; then replace each backslash-escape pair with the character it stands for, in this order: \" to a quote first, and \\ to a single backslash last, because reversing those two produces a mangled document. Then paste the result into the JSON formatter. If it indents, you had one level; if it still shows a string full of backslashes, you have a doubly-encoded payload and you repeat the pass.

the same body after one un-nest1 lines
{"method":"POST","path":"/v2/payments","headers":{"Content-Type":"application/json"},"amount":49.99,"currency":"EUR"}

117 bytes, twenty quotes with no backslashes, and now every key in it is searchable, greppable and validatable as JSON.

Escaping levels, what each costs, and the tell that identifies them
DepthWhat the line containsQuote costTell in the tool output
0A normal nested objectno escapingOutline shows keys and types; validator counts objects
1A string whose text is a JSON documentone backslash per embedded quoteOne string value marked long text; braces visible inside it
2A string whose text is a string whose text is a documentbackslash-backslash-quote per quoteThree characters per quote in the raw line; two un-nest passes
1, written defensivelyQuotes escaped as the six-character code point formsix characters per quoteSearch for the u0022 sequence; some encoders do this to survive a second pass
1 plus base64A string that is not JSON text at allno quotes visibleNo braces and no quotes in the value; see the next section

Depth is measured by how many parsers you must run, not by how ugly the line looks.

Base64 message fields

A log field that is base64 is one of three things: a binary payload somebody needed in a text file, a nested encoding used to smuggle structure through a channel that flattens everything, or an attempt at hiding something that is not hidden. All three need the same first step, which is identifying what the bytes are before you try to read them as text. Base64 characters encode bytes in order, so the first few characters of any value are a function of the first bytes of whatever it holds. Four characters decode to three bytes, which is enough for the magic numbers that matter in logs.

Leading characters, the bytes they stand for, and what you are about to look at
Field starts withDecodes toMeaningNext step
H4sI1F 8B 08gzip stream: the payload is compressed, not encryptedDecode, then inflate; a text dump of the raw bytes is meaningless
eyJ7B 22an opening brace and a quote, so JSON, and usually a JWT segmentDecode as UTF-8 and format; a dotted triple of base64url values is a JWT
iVBOR89 50 4E 47a PNG image attached to a recordReconstruct the file rather than reading it as text
/9j/FF D8 FFa JPEGReconstruct the file; check why a log carries images
UEsD50 4B 03 04a ZIP, often a jar, an Office document or an archive of dumpsExtract; look for the manifest or first entry
LS0t2D 2D 2Dthree hyphens, the start of PEM armour such as a certificate or keyInspect the label on the armour; treat key material as an incident
A-Z, digits, no patternanythingmay be a hex or base-32 string mislabelled, or a tokenTest the character set before assuming RFC 4648

A value that starts with neither a magic prefix nor a brace is still usually JSON when it decompresses, so try gzip before you try guessing.

The base64 tool reports the variant it detected rather than the one it assumed, because the standard alphabet of RFC 4648 section 4 uses plus and slash while the URL-safe alphabet of section 5 uses hyphen and underscore, and a decoder that guesses corrupts any value containing the substituted pair. It reports the decoded size, whether trailing bits are zero as they must be for a canonical encoding, the type identified from magic bytes, a hex dump of the first bytes, and whether re-encoding reproduces the input byte for byte. Order matters when encodings are stacked, and the stack in logs is almost always escape, then base64, then compression, then JSON. Take the outer escaping out first, exactly as in the previous section, and then decode.

base64 field, then the value underneath2 lines
FIELD  "payload":"eyJ1c2VyX2lkIjoiODc0MTIyMCIsInNjb3BlcyI6WyJjYXJ0OnJlYWQiLCJwYXltZW50OndyaXRlIl0sImV4cCI6MTc2MzA3MDg1NX0="
DECODE {"user_id":"8741220","scopes":["cart:read","payment:write"],"exp":1763070855}

77 bytes of payload, 104 characters of standard base64 with padding, 103 characters in base64url where padding is conventionally dropped.

Timestamps, offsets, and why sorting breaks

JSON has no date type. A timestamp in a log record is a string or a number, and the log ecosystem uses both, in at least four conventions, sometimes in the same file after a service migration. Epoch numbers are self-identifying by digit count, within the range of dates anyone is debugging in 2026. All four values below are the same instant to within a millisecond.

One instant, 2026-11-12T03:14:15.926Z, written four ways
Written asDigitsUnitWhat a naive reader does with it
2026-11-12T03:14:15.926Z24RFC 3339 with the UTC designatorParses to the instant, provided the reader accepts a millisecond fraction
179445325510Unix secondsHanded to a millisecond clock, it lands on 21 January 1970
179445325592613millisecondsHanded to a seconds clock, it lands in the year 58,935 or is rejected
179445325592600016microsecondsStill an exact integer in a double, so it survives a JSON round trip
179445325592600000019nanosecondsAbove the safe integer range: it is not the number you think it is

A tenth form shows up in old Java logs as a bare number of ticks or a formatted locale string. If the value has no Z, no offset, and is not nine to nineteen digits, it is a string you must decode with a declared layout.

The 1970 date is the most useful symptom in this whole page. A pipeline that reads seconds and hands them to an API expecting milliseconds produces events dated in January 1970, and the only reliable fix is to name the unit in the field. For nanoseconds, the largest integer that survives is 9,007,199,254,740,991, and a nanosecond epoch for the current decade is around 1.79 quintillion, five orders of magnitude past it. The subtlety is that printing the value back out does not look like damage: JavaScript renders the nearest double as 1794453255926000000, the same digits you wrote. What you have lost is resolution: every timestamp within about 128 nanoseconds of the original maps onto the same number.

Offsets, drift and why sorting breaks

RFC 3339 is a small specification, and the grammar is the whole argument: a date-time is a full date, the letter T, and a full time, and the full time includes either the UTC designator or a numeric offset. There is no production for a date-time without a zone, which means 2026-11-12T03:14:15 is not an RFC 3339 value at all. It is ISO 8601 local time, and its meaning depends on a fact the file does not record: the zone the writer was standing in.

the same instant, sorted as text5 lines
2026-11-12T02:14:15.926-01:00   -> 1794453255926
2026-11-12T03:14:15.926+00:00   -> 1794453255926
2026-11-12T03:14:15.926Z        -> 1794453255926
2026-11-12T04:14:15.926+01:00   -> 1794453255926
text sort gives them in exactly this order, 02: before 03: before 04:

Sorting by characters orders wall-clock times, not instants. Equal instants scatter across the ordering.

The version that costs an afternoon is the record whose timestamp has no offset. Parse 2026-11-12T03:14:15 on a host set to UTC+8 and you get 2026-11-11T19:14:15.000Z: the event moves eight hours into the past and, worse, across a date boundary, so it lands in the wrong day partition. Daylight saving adds the last wrinkle: a fall-back hour occurs twice in local time, so two distinct instants share one timestamp text.

Sorting failure modes, ranked by how hard they are to notice
FailureMechanismSymptom in a chartTest that catches it
Mixed unitsAny milliseconds value exceeds any seconds value, so all seconds records precede all millisecond onesTwo blocks with a gap between themTake the digit count of every epoch value in the column
Text sort on epochsTen-digit strings compare before thirteen-digit stringsRecent events at the bottom of an ascending sortCast to a number before sorting; compare against a byte-length histogram
Text sort on offsetsWall-clock characters order the string, not the instantA foreign region interleaved wronglyNormalise every value to Z form, then sort
Zone-less local timeThe parser supplies its own zoneEvents shifted by a constant, sometimes across a day boundaryParse one record, print it back with the zone, compare
Ingest time masqueradingA pipeline that cannot parse the declared field stamps arrival insteadEvents clustered on the collector clock, sharp boundaries at deploysCompare the timestamp field against the record arrival time
DST foldOne local hour happens twiceDuplicate timestamps in a single host logSort and look for a repeated hour at the transition

Nothing on this site sorts records. Sort in the system that will hold the records, after normalising the timestamp to one unit and one zone.

Identifiers through a double and giant payloads

Log records are full of identifiers that happen to be digits: trace IDs, snowflake IDs, span IDs, user IDs issued by an ORM, database log sequence numbers, Kafka offsets. When one of those arrives as a JSON number rather than a string, it is no longer an identifier. It is a floating-point approximation of one, and the approximation is silent. The largest integer a double holds exactly is 9,007,199,254,740,991, and a nineteen-digit snowflake such as 1854163187648970752 leaves a JSON round trip as 1854163187648970800: the last two digits are invented. Worse, three consecutive trace IDs from the same service all print as the same number. Put them through a parser, then deduplicate, and you have one event where the system emitted three.

Precision damage by identifier family
IdentifierTypical digitsThrough a doubleCorrect handling
Snowflake and similar time-ordered ids18 to 19Rounded; adjacent ids collapseString in JSON; 64-bit integer in the database
W3C trace context trace id32 hex charactersNot a number at allString; never parse it as a number, ever
PostgreSQL bigserialup to 19Rounded above 16 digitsString in transit; bigint in storage
Kafka offsetup to 19RoundedString in transit; int64 in the consumer
Microsecond epoch16Exact until about 2255Number is safe; document the unit
Nanosecond epoch19Rounded to roughly the nearest 128 nsSplit into a seconds field and a nanosecond field, or use a typed integer column

The rule that prevents all of this: a value is an identifier when it is compared for equality and never added up. Identifiers are strings, in logs and in JSON.

The strict path and the repair path in our own pages behave differently here. Paste {"id":9007199254740993} into the JSON validator and it reports valid with one unsafe-integer warning. The formatter prints from the parse tree, so the digit string survives formatting unchanged. Repair mode is the opposite: it rebuilds the document from a JavaScript value, and that is where the digits move. To inspect a record without touching its values, use the strict formatter. To make a file queryable, wrap the lines by hand and format strictly. Reach for repair mode only when you have confirmed the input has no long numeric identifiers.

Giant single-line payloads

A log line can be enormous and still be one line: a serialized request body, a base64 image, a feature-flag evaluation containing the whole configuration. The moment a record is bigger than what your clipboard, your terminal and your browser tab are comfortable with, the debugging method has to change from reading to instrumenting. Everything here runs in the tab, so a four-megabyte line is comfortable; a hundred megabytes of NDJSON is not, and the failure mode is a frozen tab rather than an error message. The parser is a scanner over the whole string, so a huge valid record costs one pass, and the error report is designed for the case where you cannot read the line: the validator gives line, column and offset, then a one-line excerpt of roughly twenty-five characters before and ninety after the fault.

Truncation and size symptoms, and the diagnostic that names them
Symptom in the reportLikely causeHow to confirmWhat to do
Input ended before a value was foundThe line was cut by a copy, a proxy buffer or a field length limitThe offset equals the last byte of the inputRe-export the record; the tail does not exist in the file
Object is missing its closing braceA truncated response body pasted aloneBrace depth never returns to zeroGet the full line, or repair and inspect the partial object
Unexpected closing bracket after the top-level valueAn extra brace at the end, or two records jammed togetherOffset points past the end of the first valueSplit at the brace pair and validate each piece
Nesting is deeper than 200 levelsA recursive or self-referencing structure logged as dataThe depth counter in the statsThe scanner refuses; flatten at the writer
More than one value at the top levelNDJSON, an orphan tail after a complete record, or a footer lineCompare the offset with the start of line 2See the sections on framing and torn lines
Tab unresponsive after pastingPayload beyond comfortable memory, not a data problemThe byte count in the input paneWork on one record at a time, or use a streaming tool

For the shipping and archiving question, the JSON minifier measures the answer: byte savings against the input, and a gzip comparison of both sides.

Once a payload parses, the outline is the tool you want rather than the formatted output. An object line shows its key count with the first twelve members expanded, an array shows its item count and descends only into the first element, and any string longer than forty characters is flagged as long text. That marker is how you find escaped bodies and base64 blobs in a wide record without reading it.

Stack traces and CSV exports

Exceptions are multi-line and JSON strings are not, so a stack trace in a log record is always an escaped thing: a single string value containing backslash-n sequences where the line breaks used to be. A trace with four line feeds and no quotes is 326 bytes raw and 332 bytes as a JSON string value, because each line feed becomes the two-character escape and the value gains its wrapping quotes. Multiply that by a forty-frame stack and the field is routinely larger than the rest of the record combined.

a trace as it is stored1 lines
{"ts":"2026-11-12T03:14:16.884Z","level":"error","svc":"checkout","msg":"db failure","stack":"Error: connect ETIMEDOUT 10.4.18.77:5432\n    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)\n    at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17)\n    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)\n    at Pool._acquire (file:///app/node_modules/pg-pool/index.js:45:12)"}

One string, four escape sequences, and every frame of the failure compressed into a single greppable line. This is the correct way to log a trace, and the least readable.

Reading it properly means turning the escapes back into line breaks. The frames are ordered innermost first, so the interesting line is usually the last application frame rather than the first. Two things to check in the record itself: if the exception was serialised by a generic encoder, the cause chain may be nested one level deeper and escaped again, giving you an escape inside an escape — the six-character sequence backslash-backslash-n in a raw file means a newline inside a string inside a string, and you need two un-escape passes.

CSV exports of log queries

Sooner or later a log query becomes a spreadsheet: an alert export, a support attachment, a column of request IDs pasted into a sheet for counting. The exporter flattens your records into columns, and the file that comes back is a text format with no types, which means every typing decision you made in the pipeline is now somebody else's guess. The first trap is the delimiter inside a value: log exports love columns that hold JSON, label sets or user agents, and all three contain commas. The second is column typing: an identifier column that looks numeric is the classic casualty. The third is the timestamp column, which arrives as text and stays as text: sorting it is the wall-clock ordering described earlier. The fourth is the double click: opening a CSV in a spreadsheet re-types columns by content, keeps about fifteen significant digits, and turns free text beginning with equals into a formula.

Log-export columns and the safe handling for each
ColumnWhat it looks likeWhat a naive reader doesWhat to do
@timestampRFC 3339 text, sometimes with an offsetText sort; date filter with an assumed zoneNormalise to Z before exporting, and keep the original in a second column
trace_id32 hex charactersNothing dangerous, unless it is numeric-lookingKeep as text; never let it be a number
span_id, user_id18 to 19 digitsRounded by a double, by Excel, by the readerQuote on write; force text on import
labels or tagsJSON with commas and quotesSplits into extra fields if unquotedExport flattened columns, or quote and re-parse
messageFree text, sometimes with newlinesA row count that lies, because quoted newlines are real rowsCheck the multi-line cell count before trusting the row count
duration_ms12.50, 3, 1e3Retyped, reformatted, trailing zeros droppedType it as a number on purpose, with a documented unit
status200, 404, padded codesLeading zeros lostForce text

A spreadsheet tops out at 1,048,576 rows and 16,384 columns, so a log export aimed at a grid needs a row budget decided before the query runs.

Which page, in which order

The log-file runbook

Everything above collapses into a sequence with a page attached to each step. Work down it once per incident and the whole file becomes describable in about five minutes.

The order of operations, and the page to open at each one
#StepOpenWhat you are reading forStop condition
1Confirm the framingJSON formatterWhether the newline-delimited note appears, and the record count it statesNo note means a torn or multi-line record: go to step 4
2Get the exact fault positionsJSON validatorLine, column and offset for every error, plus the excerpt window around the firstAny error is a reason not to analyse yet
3Wrap without convertingJSON formatterByte arithmetic: brackets plus commas, records unchangedA diff of any record before and after is empty
4Hunt the fragmentsJSON validatorUnterminated strings and unquoted keys at column 1 in a split fileEvery kept record validates alone
5Count records twiceValidator stats, then your line countRecords in the array root against non-blank linesThe two numbers are equal
6Audit the warningsJSON validatorunsafe-integer, duplicate-key and byte-order-mark countsEach warning is explained or fixed at the writer
7Un-nest escaped bodiesJSON formatterLong-text flags in the outline, then the decoded value pasted backThe inner document indents
8Decode encoded fieldsBase64Variant detected, magic bytes, decoded size, canonical or notYou can name the payload type
9Normalise timestampsSpreadsheet or query engine, guided by the tables aboveOne unit, one zone, offsets converted to the UTC designatorSorting by the column matches sorting by the instant
10Verify a large payloadJSON minifierByte savings and the gzip comparison for the shipping decisionRound-trip check passes
11Return from a CSV exportCSV to JSONDelimiter reason, field counts, per-column typesNo errors, and no mixed column you did not choose
12Read the specification againJSON guideThe one-value rule, number limits, duplicate namesYou can explain the error to someone else

Steps 1 to 6 are read-only and change nothing. Steps 7 and 8 are transforms and each needs a re-validation. Step 12 is the one that makes the next incident shorter.

NDJSON / JSON Lines
One JSON value per line, line feed as the record delimiter, UTF-8 encoding. A convention documented by its own spec site, not an RFC, and invalid as a single JSON text.
Top-level value
The one thing RFC 8259 permits in a JSON text. Anything after it, including a second record or an orphan fragment, is an error reported at that byte offset.
Torn line
Two writers interleaved into one line. Shows up as a head with an unterminated string and a tail that starts with unquoted text.
Double-encoded field
A string whose characters are a JSON document, produced by serialising twice. Costs one backslash per embedded quote and requires one parse per level.
JSON text sequence
RFC 7464 framing: each element prefixed by the record separator U+001E and terminated by a line feed, so a truncated tail is detectable.
Safe integer
9,007,199,254,740,991. Above it a JSON number is an approximation, so identifiers and nanosecond clocks are strings or typed integers, never numbers.
Repair mode
Lenient re-serialisation that collects several top-level values into one array. Useful, but values pass through a double and duplicate keys collapse.
Structure outline
A container-level walk with key counts, item counts and a long-text flag, which is how you find buried payloads in a wide record without printing it.

Answers

Questions about reading structured logs

Why is my log file "invalid JSON" when every line parses?

Because RFC 8259 defines a JSON text as one value: the grammar is ws value ws, with no production for a second value after it. A file of one-object-per-line records is newline-delimited JSON, which is a convention documented by its own spec site, not a variant of JSON. The JSON validator is right to reject it; the fix is to wrap the lines in brackets separated by commas.

How do I turn NDJSON into a JSON array without dropping records?

Wrap, do not re-encode. Put a [ before the first line, replace each line feed between records with a comma, and put a ] after the last. Then open the JSON formatter and compare two numbers: the record count in the stats grid must equal your expected line count, and the report of further errors must be empty.

What does a torn log line look like?

Two fragments instead of one record. The head ends inside a string, so the validator reports "String is never closed" at the opening quote; the tail begins with leftover text and is reported as an unquoted key at column 1. Neither line is a record. The file changed underneath you — two writers interleaved into one line.

Why does sorting my log lines by the timestamp field give the wrong order?

Three separate causes. Text sorting compares characters, so an offset-bearing timestamp sorts by wall-clock time, not by instant. Numeric sorting with mixed units compares magnitudes, so every seconds value precedes every milliseconds value. And a timestamp with no offset is local wall-clock time: parse one on a machine in another zone and the instant moves by the whole offset.

Related

Next up

Sources and further reading

  1. [1]RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format — Section 2, the grammar `JSON-text = ws value ws`, which is why a file of records is not one JSON text; section 6 on number range, precision and the ban on leading zeros; section 4 on member names; section 7 on escaping control characters; and section 8, which requires UTF-8 and forbids a byte order mark. (accessed November 12, 2026)
  2. [2]RFC 7464: JSON Text Sequences — The one standards-track way to put several JSON texts in a stream: each element is prefixed with the record separator RS, `%x1E`, and terminated with a line feed; the media type is application/json-seq; and a sequence whose final element is cut off is still parsable. (accessed November 12, 2026)
  3. [3]RFC 3339: Date and Time Format — The Internet-profile grammar `date-time = full-date "T" full-time`, where full-time carries either the UTC designator or a numeric offset, hours run 00-23, and an offset of -00:00 is explicitly different in meaning from Z. (accessed November 12, 2026)
  4. [4]JSON Lines (jsonlines.org) — The convention site for newline-delimited records: each line is a valid JSON value, the terminator is a line feed, encoding must be UTF-8, empty lines are not allowed, and the media type is described as not yet standardized. (accessed November 12, 2026)
  5. [5]Elasticsearch Reference: Bulk API — A production example of newline-delimited JSON as a wire format: literal newline characters as delimiters, the final line must end with a newline, the Content-Type is application/json or application/x-ndjson, and actions must not be pretty printed. (accessed November 12, 2026)