Skip to content

LLM token counter and RAG text chunker

Count exact BPE tokens offline, inspect color-coded token splits, and chunk long documents for RAG pipelines without sending prompts over the network.

Runs BPE tokenization in a local Web Worker.

How to use LLM token counter

  1. Paste text or load a document. Enter prompt copy into Prompt, Markdown, or document text, click Load sample prompt, or drop a .txt, .md, .json, .yaml, .csv, or code file.
  2. Select an encoding and model rate. Choose a Model preset or BPE encoding between o200k_base and cl100k_base, then adjust per-million token rates and Expected output tokens.
  3. Configure RAG chunking parameters. Pick Markdown heading aware or Fixed token window under Chunk strategy, set Chunk tokens and Overlap, and click Count tokens & chunk.
  4. Inspect splits and export chunks. Toggle Show BPE token IDs in the inspector, verify context fit bars, then click Download JSONL, Download CSV, or Download ZIP bundle.

Everything runs in this browser tab. Nothing you paste or open is uploaded or logged, so it's safe for keys, tokens and private data.

Why do o200k_base and cl100k_base token counts differ?

OpenAI doubled the byte pair encoding merge vocabulary from 100,277 entries in cl100k_base to 200,019 entries in o200k_base, allowing newer reasoning and multimodal models to pack longer subwords into one token. Checking prompts in this local LLM token counter immediately exposes lower token totals on structured JSON, source code, and non-English scripts.

Paste raw prompts or drop .txt, .md, .json, .yaml, .csv, .ts, .py, or .sql files, then pick a Model preset or BPE encoding. All tokenization runs locally inside a background Web Worker without uploading prompts or API drafts.

Model preset BPE encoding Context window Input rate (USD / 1M) Output rate (USD / 1M)
GPT-4o o200k_base 128,000 tokens $2.50 $10.00
GPT-4.1 o200k_base 1,047,576 tokens $2.00 $8.00
o3 / o4-mini o200k_base 200,000 tokens $2.00 $8.00
GPT-4o mini o200k_base 128,000 tokens $0.15 $0.60
GPT-4 / GPT-3.5 cl100k_base 128,000 tokens $10.00 $30.00

These encoding rules follow the merge tables published in the OpenAI tiktoken BPE tokenizer repository. When auditing editorial prose length alongside prompt budgets, pair token counts with human word counts without BPE subword splits in the same browser session.

Inspecting BPE token boundaries and multi-byte UTF-8 splits

Summary numbers alone hide why two similar strings produce wildly different token totals when punctuation, whitespace, or non-ASCII scripts change. Enabling the Token boundary inspector renders alternating color chips across every token so subword breaks appear cleanly.

Color-coded token spans and raw BPE IDs

The inspector displays up to 1,200 tokens from the start of your text across five rotating color swatches, showing spaces as middle dots (·), newlines as return arrows (↵), and tabs as right arrows (→). Toggling Show BPE token IDs prints the exact #<id> integer beside each token chip.

Inspecting those visual splits catches bloated indentation, repeated delimiters, and unminified JSON whitespace inside system prompts. Before embedding API schemas in few-shot instructions, compact raw fixtures using our strict JSON payload formatter and validator to compare formatted versus single-line token costs.

Incomplete UTF-8 byte tokens and encoding metrics

Byte-level BPE operates on UTF-8 byte sequences standardized under the IETF RFC 3629 UTF-8 encoding standard rather than full Unicode characters. Multi-byte emoji, accented letters, and non-Latin glyphs frequently split across two or three consecutive BPE tokens.

Whenever decoding an isolated token produces U+FFFD (an incomplete UTF-8 fragment), the span falls back to its hexadecimal ID ([0x...]) instead of showing broken text. Summary cards also compute exact charsPerToken and bytesPerToken metrics (such as 3.8 bytes per token) alongside character, word, and line counts.

Splitting Markdown documents into overlapping RAG chunks

Retrieval-augmented generation loses factual precision when fixed byte offsets slice headings away from their supporting paragraphs or split code snippets in half. Choosing Markdown heading aware under Chunk strategy keeps document hierarchy intact before applying token caps.

Markdown heading-aware packing vs fixed windows

Markdown-aware splitting scans lines outside triple-backtick (```) and triple-tilde (~~~) code fences for # through ###### headings. Short sections at or below Chunk tokens (default 400 tokens, UI range 32 to 16,384 tokens) stay intact as single chunks carrying their exact heading title.

Switching Chunk strategy to Fixed token window ignores Markdown headings and slices the entire encoded token array uniformly from start to finish. If source articles arrive as copied web pages or Google Docs snippets, convert them with the HTML to clean Markdown converter first so heading boundaries survive intact.

Preserving fenced code blocks and token overlap

Oversized Markdown sections split along blank lines into paragraph and code-block units without breaking fenced code blocks mid-example. Each unit packs greedily into the current chunk until adding the next block exceeds Chunk tokens.

If an individual paragraph or code block itself exceeds Chunk tokens, the worker flushes prior blocks and applies a sliding window across that oversized unit. Setting Overlap (default 40 tokens, UI range 0 to 2,048 tokens) prepends trailing tokens from each chunk onto the next segment so boundary context stays connected.

Tuning chunk size and overlap for vector retrieval

Embedding benchmarks typically perform best with 256 to 512 token chunks and 10% to 15% overlap for targeted Q&A lookup. Larger 800 to 1,200 token chunks preserve broader policy context when prompts retrieve full runbooks or compliance clauses.

Tip: Keep Overlap strictly smaller than Chunk tokens (for instance 40 overlap tokens on a 400 token chunk). The splitter clamps overlap at chunkSize - 1 so sliding windows always advance by at least one token per step.

Estimating API token budgets and context window fit

Cost projections multiply counted prompt tokens against Input price ($ / 1M tokens) and multiply Expected output tokens (default 500) against Output price ($ / 1M tokens). As outlined in the OpenAI Cookbook token counting notebook, exact client-side counting prevents surprise truncation errors or budget overruns on large batch jobs.

Selecting Custom rate lets you enter per-million input and output prices for Anthropic, Gemini, or self-hosted endpoints while keeping o200k_base or cl100k_base counts active. Tiny prompt totals format down to five decimal places ($0.00012) so per-call micro-costs remain legible during prompt tuning. Comparing o200k_base against legacy cl100k_base counts on long JSON schemas reveals immediate cost reductions before deploying batch workflows.

The Model context window utilization grid checks input length against 128,000 tokens (GPT-4o / Mini), 200,000 tokens (o3 / Claude), and 1,047,576 tokens (GPT-4.1 / Gemini). Leave headroom below 85% utilization for system prompts, tool schemas, and completion tokens so long replies never hit context truncation mid-stream.

What export formats does the RAG chunker produce?

Generated RAG chunks export straight from browser memory in three pipeline-ready formats: newline-delimited JSONL (rag-chunks.jsonl), spreadsheet-safe CSV (rag-chunks.csv), and a DEFLATE-compressed ZIP archive (rag-chunks.zip). Every exported row preserves sequential chunk_index, section heading, exact start_token and end_token offsets, character count, and decoded chunk text.

  • Download JSONL: Writes one JSON object per line with chunk_index, heading, start_token, end_token, token_count, char_count, and text keys ready for embedding scripts. Load the downloaded file afterwards to validate exported RAG JSONL chunk files before indexing.
  • Download CSV: Adds a UTF-8 byte order mark for reliable Excel opening and applies MITRE CWE-1236 CSV formula neutralization, prefixing cells that begin with =, +, -, or @ with a single apostrophe.
  • Download ZIP bundle: Packs zero-padded Markdown files (rag-chunks/chunk-001.md, chunk-002.md) alongside a combined chunks.jsonl index using level 6 DEFLATE compression via fflate inside your browser.

Questions

What is the difference between o200k_base and cl100k_base encodings?

OpenAI expanded the BPE merge vocabulary from 100,277 entries in cl100k_base to 200,019 entries in o200k_base for GPT-4o, GPT-4.1, and o-series models. Because o200k_base merges longer punctuation, JSON, and non-English byte runs into single tokens, token counts drop noticeably relative to cl100k_base.

How does Markdown heading-aware RAG chunking preserve code blocks?

The chunker tracks fenced blocks opened by triple backticks or tildes so comment hashes inside code snippets never act as section breaks. Sections under Chunk tokens stay whole, while larger sections group blank-line paragraphs and complete code blocks before falling back to sliding overlap slices.

Why do single BPE token spans sometimes display hex codes like [0x...]?

Byte pair encoding operates on raw UTF-8 bytes rather than whole Unicode code points, so emoji, CJK characters, or accented letters often split across two or three consecutive tokens. When decoding one token produces the Unicode replacement character, the inspector prints its hexadecimal token ID safely.

How are API request costs and context window percentages calculated?

Input cost multiplies your counted tokens by Input price divided by 1,000,000, while output cost applies Output price to Expected output tokens. Context utilization divides counted input tokens against 128,000, 200,000, and 1,047,576 token limits, highlighting red once input alone exceeds capacity.

What happens when a single paragraph or code block exceeds Chunk tokens?

When one standalone paragraph or fenced code block contains more tokens than Chunk tokens, Markdown heading-aware mode flushes accumulated blocks first and slices that oversized unit with a sliding token window. Adjacent sub-chunks keep your Overlap token count and retain the parent section heading.

Which file formats can I drop or export from the token counter?

You can drop local .txt, .md, .markdown, .json, .yaml, .yml, .csv, .ts, .py, and .sql text files without uploading prompts anywhere. Generated RAG chunks export directly as rag-chunks.jsonl, UTF-8 BOM rag-chunks.csv, or a DEFLATE-compressed rag-chunks.zip bundle containing numbered Markdown files.

Updated