Why do o200k_base and cl100k_base token counts differ?
OpenAI doubled the byte pair encoding merge vocabulary from 100,277 entries in cl100k_base to 200,019 entries in o200k_base, allowing newer reasoning and multimodal models to pack longer subwords into one token. Checking prompts in this local LLM token counter immediately exposes lower token totals on structured JSON, source code, and non-English scripts.
Paste raw prompts or drop .txt, .md, .json, .yaml, .csv, .ts, .py, or .sql files, then pick a Model preset or BPE encoding. All tokenization runs locally inside a background Web Worker without uploading prompts or API drafts.
| Model preset | BPE encoding | Context window | Input rate (USD / 1M) | Output rate (USD / 1M) |
|---|---|---|---|---|
GPT-4o |
o200k_base |
128,000 tokens |
$2.50 |
$10.00 |
GPT-4.1 |
o200k_base |
1,047,576 tokens |
$2.00 |
$8.00 |
o3 / o4-mini |
o200k_base |
200,000 tokens |
$2.00 |
$8.00 |
GPT-4o mini |
o200k_base |
128,000 tokens |
$0.15 |
$0.60 |
GPT-4 / GPT-3.5 |
cl100k_base |
128,000 tokens |
$10.00 |
$30.00 |
These encoding rules follow the merge tables published in the OpenAI tiktoken BPE tokenizer repository. When auditing editorial prose length alongside prompt budgets, pair token counts with human word counts without BPE subword splits in the same browser session.
Inspecting BPE token boundaries and multi-byte UTF-8 splits
Summary numbers alone hide why two similar strings produce wildly different token totals when punctuation, whitespace, or non-ASCII scripts change. Enabling the Token boundary inspector renders alternating color chips across every token so subword breaks appear cleanly.
Color-coded token spans and raw BPE IDs
The inspector displays up to 1,200 tokens from the start of your text across five rotating color swatches, showing spaces as middle dots (·), newlines as return arrows (↵), and tabs as right arrows (→). Toggling Show BPE token IDs prints the exact #<id> integer beside each token chip.
Inspecting those visual splits catches bloated indentation, repeated delimiters, and unminified JSON whitespace inside system prompts. Before embedding API schemas in few-shot instructions, compact raw fixtures using our strict JSON payload formatter and validator to compare formatted versus single-line token costs.
Incomplete UTF-8 byte tokens and encoding metrics
Byte-level BPE operates on UTF-8 byte sequences standardized under the IETF RFC 3629 UTF-8 encoding standard rather than full Unicode characters. Multi-byte emoji, accented letters, and non-Latin glyphs frequently split across two or three consecutive BPE tokens.
Whenever decoding an isolated token produces U+FFFD (an incomplete UTF-8 fragment), the span falls back to its hexadecimal ID ([0x...]) instead of showing broken text. Summary cards also compute exact charsPerToken and bytesPerToken metrics (such as 3.8 bytes per token) alongside character, word, and line counts.
Splitting Markdown documents into overlapping RAG chunks
Retrieval-augmented generation loses factual precision when fixed byte offsets slice headings away from their supporting paragraphs or split code snippets in half. Choosing Markdown heading aware under Chunk strategy keeps document hierarchy intact before applying token caps.
Markdown heading-aware packing vs fixed windows
Markdown-aware splitting scans lines outside triple-backtick (```) and triple-tilde (~~~) code fences for # through ###### headings. Short sections at or below Chunk tokens (default 400 tokens, UI range 32 to 16,384 tokens) stay intact as single chunks carrying their exact heading title.
Switching Chunk strategy to Fixed token window ignores Markdown headings and slices the entire encoded token array uniformly from start to finish. If source articles arrive as copied web pages or Google Docs snippets, convert them with the HTML to clean Markdown converter first so heading boundaries survive intact.
Preserving fenced code blocks and token overlap
Oversized Markdown sections split along blank lines into paragraph and code-block units without breaking fenced code blocks mid-example. Each unit packs greedily into the current chunk until adding the next block exceeds Chunk tokens.
If an individual paragraph or code block itself exceeds Chunk tokens, the worker flushes prior blocks and applies a sliding window across that oversized unit. Setting Overlap (default 40 tokens, UI range 0 to 2,048 tokens) prepends trailing tokens from each chunk onto the next segment so boundary context stays connected.
Tuning chunk size and overlap for vector retrieval
Embedding benchmarks typically perform best with 256 to 512 token chunks and 10% to 15% overlap for targeted Q&A lookup. Larger 800 to 1,200 token chunks preserve broader policy context when prompts retrieve full runbooks or compliance clauses.
Tip: Keep Overlap strictly smaller than Chunk tokens (for instance
40overlap tokens on a400token chunk). The splitter clamps overlap atchunkSize - 1so sliding windows always advance by at least one token per step.
Estimating API token budgets and context window fit
Cost projections multiply counted prompt tokens against Input price ($ / 1M tokens) and multiply Expected output tokens (default 500) against Output price ($ / 1M tokens). As outlined in the OpenAI Cookbook token counting notebook, exact client-side counting prevents surprise truncation errors or budget overruns on large batch jobs.
Selecting Custom rate lets you enter per-million input and output prices for Anthropic, Gemini, or self-hosted endpoints while keeping o200k_base or cl100k_base counts active. Tiny prompt totals format down to five decimal places ($0.00012) so per-call micro-costs remain legible during prompt tuning. Comparing o200k_base against legacy cl100k_base counts on long JSON schemas reveals immediate cost reductions before deploying batch workflows.
The Model context window utilization grid checks input length against 128,000 tokens (GPT-4o / Mini), 200,000 tokens (o3 / Claude), and 1,047,576 tokens (GPT-4.1 / Gemini). Leave headroom below 85% utilization for system prompts, tool schemas, and completion tokens so long replies never hit context truncation mid-stream.
What export formats does the RAG chunker produce?
Generated RAG chunks export straight from browser memory in three pipeline-ready formats: newline-delimited JSONL (rag-chunks.jsonl), spreadsheet-safe CSV (rag-chunks.csv), and a DEFLATE-compressed ZIP archive (rag-chunks.zip). Every exported row preserves sequential chunk_index, section heading, exact start_token and end_token offsets, character count, and decoded chunk text.
- Download JSONL: Writes one JSON object per line with
chunk_index,heading,start_token,end_token,token_count,char_count, andtextkeys ready for embedding scripts. Load the downloaded file afterwards to validate exported RAG JSONL chunk files before indexing. - Download CSV: Adds a UTF-8 byte order mark for reliable Excel opening and applies MITRE CWE-1236 CSV formula neutralization, prefixing cells that begin with
=,+,-, or@with a single apostrophe. - Download ZIP bundle: Packs zero-padded Markdown files (
rag-chunks/chunk-001.md,chunk-002.md) alongside a combinedchunks.jsonlindex using level6DEFLATE compression viafflateinside your browser.