Why use JSONL or NDJSON instead of a giant JSON array?
Newline-delimited records parse independently line by line, isolating trailing commas or unclosed braces without aborting the entire dataset. Reading one object per row keeps multi-megabyte log streams and fine-tuning corpora manageable in browser memory while revealing line numbers immediately.
As defined in the JSON Lines on the Web specification, each non-empty line contains one self-contained UTF-8 object separated by a newline delimiter. Our JSONL and NDJSON viewer strips leading UTF-8 byte order marks (\uFEFF), normalizes CRLF and LF endings, and counts blank lines separately without misreporting trailing newlines as broken rows.
Traditional editors choke when one missing brace invalidates a 25 MB JSON array, hiding everything past that syntax fault. When isolated records need structural pretty-printing or syntax fixes before re-assembly, try browser JSON syntax repair without server uploads on individual objects, then paste them back into your corpus.
All line splitting, schema validation, BPE counting, and ZIP archiving execute locally inside a dedicated Web Worker (dev.worker.ts). Zero training prompts, telemetry logs, or file names cross the network during inspection.
Validating OpenAI, Anthropic, and generic JSONL schemas
Selecting Auto-detect schema under Dataset validation schema inspects the first 25 valid JSON objects to match conversational LLM training formats or fall back to generic NDJSON. You can override auto-detection at any time and trigger Validate JSONL across every line.
OpenAI Chat fine-tuning message rules
The OpenAI Chat validator enforces conversational rules documented in the OpenAI supervised fine-tuning guide. Every line must hold a non-empty messages array whose entries use recognized roles (system, developer, user, assistant, tool, or function).
Each message requires string or structured-content array data, except assistant turns carrying tool_calls or function_call objects. Every record also requires at least one assistant turn so training never ingests prompt-only conversations.
Anthropic Messages alternating turn checks
For Claude evaluation and tuning datasets, the Anthropic Messages preset validates turn ordering matching the Anthropic Messages API schema reference. An optional top-level system key accepts either a plain string or non-empty block array rather than embedding system instructions inside messages.
Conversations must start with a user turn, alternate strictly between user and assistant on every consecutive step, and end on an assistant reply. Misordered turns or duplicate consecutive roles trigger exact row-level errors immediately.
Prompt-completion pairs and generic NDJSON mode
Legacy completion corpora work under Prompt / Completion, requiring non-empty prompt and completion fields or instruction and output pairs. Choosing Generic NDJSON checks valid object or JSON value syntax per line for structured application logs and webhook events. If source records live in tabular sheets first, parse CSV spreadsheets into JSON records before running schema checks.
| Validation preset | Required root fields | Role / turn constraint | Blank / syntax behavior | Primary use case |
|---|---|---|---|---|
Auto-detect schema |
Inspects first 25 objects |
Selects OpenAI, Anthropic, or pair mode | Skips blank lines cleanly | Mixed unfamiliar datasets |
OpenAI Chat |
Non-empty messages array |
Six valid roles; needs assistant |
Rejects missing content |
GPT fine-tuning runs |
Anthropic Messages |
messages + optional system |
Strict user to assistant alternation |
Flags consecutive identical roles | Claude conversation sets |
Prompt / Completion |
prompt+completion or instruction+output |
Rejects blank input/output strings | Flags syntax errors by line | Legacy instruction corpora |
Generic NDJSON |
Any valid JSON line value | No chat role requirements | Counts syntax errors only | Log streams and telemetry |
Token estimation and record-level error filtering
Validating syntax and schema structure runs alongside summary metrics in four live cards: Detected Schema, Valid JSON / Schema, Dataset Tokens, and Top Roles / Keys (tracking the 20 most frequent roles or top-level object keys).
Exact BPE counting vs fast multi-megabyte heuristics
To balance precision with speed in browser memory, records up to 1,500 lines run through exact o200k_base byte pair encoding inside the worker. Once a dataset exceeds 1,500 lines, the estimator switches to a calibrated character heuristic (3.8 characters per token) so 50 MB log dumps stay interactive under 500 ms.
When small system-prompt templates need granular token boundary color highlights or custom 128K context pricing comparisons, open them in our dedicated token counter. Combining corpus validation with exact prompt budgeting keeps API batch pipelines predictable.
Filtering malformed lines and exporting triage reports
Every line checked against the IETF RFC 8259 JSON specification feeds into Record inspector, which displays up to 250 filtered rows at a time and truncates raw preview strings at 140 characters. Toggle All rows, Errors / issues, or Valid only to isolate trailing commas or schema mismatches fast.
Clicking View opens a pretty-printed JSON modal with a Copy JSON button for instant repair. QA reviewers can also download jsonl-validation-report.csv (framed with a UTF-8 BOM and MITRE CWE-1236 formula neutralization) or jsonl-validation-report.xlsx (JSONL Report sheet) via Export CSV report and Export Excel report.
How does deterministic train and validation splitting work?
A seeded mulberry32 pseudorandom generator paired with Fisher-Yates shuffling assigns identical rows to training and holdout slices whenever your seed number stays fixed. Choosing a training percentage allocates exact partition counts before packing output files.
Adjust Train ratio (%) between 10% and 99% (80% default in the UI) and set Deterministic seed (42 default) before clicking Preview split counts to confirm row counts without writing files first. Unchecking Shuffle before split keeps chronological line order intact for time-series telemetry logs.
When ready, export standalone files with Download train.jsonl and Download val.jsonl, or click Download train + val ZIP to package dataset-train.jsonl and dataset-val.jsonl inside dataset-split.zip using level 6 DEFLATE compression. Checking Keep schema-valid lines only ensures broken records never contaminate fine-tuning jobs.
Tip: Enable
Keep schema-valid lines onlyand record yourDeterministic seedvalue in training run logs so teammates can reproduce exact holdout evaluation slices later.
Processing large NDJSON corpora offline in browser memory
Every stage from file drop to archive generation operates strictly on local bytes in browser RAM (Uint8Array slices inside dev.worker.ts), avoiding server uploads that leak proprietary customer payloads or PII telemetry. Because line splitting, token estimation, and JSON parsing execute off the main UI thread inside our JSONL viewer, even multi-megabyte webhook dumps keep page scrolling responsive.
Filtering previews inside Record inspector at 250 visible DOM rows prevents layout stalls even when inspecting tens of thousands of NDJSON records. Blank separator lines and trailing newlines are tallied accurately without mutating source text or flagging false syntax failures at end-of-file.
Teams preparing private evaluation logs or internal agent transcripts retain deterministic train/validation bundles straight in Downloads folders without external SaaS accounts. If downstream analytics pipelines need column-oriented storage instead of row-oriented NDJSON logs, inspect matching analytical warehouse outputs using our columnar Apache Parquet dataset inspector locally as well.