See Exactly Where Your Document Gets Chunked

Why chunking quietly decides RAG quality

Retrieval-augmented generation only ever shows the model the handful of passages that retrieval picked. Those passages are your chunks, so every choice you make when splitting documents, the chunk size, the overlap and the strategy, sets a ceiling on how good the answers can be. A chunk that stops halfway through a definition embeds well and answers badly. A heading stranded at the bottom of one chunk while its body sits in the next means the body is retrieved without the context that says what it is about. A chunk that straddles three topics matches none of them strongly.

Most teams choose these values once, blind, from a framework default, and never see the result. This tool shows it. Paste a real document, move the sliders, and watch where the boundaries land before any of it goes near an embedding API.

How the four strategies split a document

Fixed size (characters) and Fixed size (tokens) cut windows of exactly the configured length with a stride of size minus overlap. They ignore the text completely, which makes them fast, predictable, and the baseline every paper compares against. The token version encodes the document once with the chosen tokenizer and cuts on token boundaries, the same way a TokenTextSplitter does, so every chunk but the last is exactly the size you asked for.

Sentence-aware finds sentence boundaries with the browser's Intl.Segmenter, which follows the Unicode text segmentation rules and handles lowercase continuations after a full stop, closing quotes and CJK punctuation, then packs whole sentences up to the limit. The last sentences of a chunk are carried into the next one as overlap, so overlap is always made of complete sentences. A single sentence longer than the limit is split at the limit and the tool says so.

Recursive is the default because it is modelled on the splitter LangChain and LlamaIndex ship. It splits on the first separator in the cascade, paragraph break, then line break, then sentence, then word, merges the pieces that fit up to the limit, and splits any piece that does not fit again with the next separator. The sentence step is this tool's addition: LangChain's default cascade goes from line breaks straight to spaces and then to single characters, so its boundaries will not match exactly. A chunk boundary is forced on both sides of an oversized paragraph, which is why the recursive run usually has a few more chunks than the fixed run on the same text: it refuses to fill a chunk across a paragraph break.

["\n\n", "\n", sentence, " "]

1. paragraphs stay whole when they fit
2. a long paragraph splits at line breaks (lists, code)
3. a long line splits at sentence ends
4. a long sentence splits between words
5. a single oversized word is cut at the limit

Switch between the strategies on the same document to see them disagree. The example document was written so that the fixed strategies cut its YAML block and its table in half while the recursive strategy keeps both intact.

Choosing chunk size and overlap

There is no universal right answer; the best values depend on how your documents are written and what people ask. These rules of thumb are where to start, not where to stop.

• Start at 500 tokens with 10 percent overlap. It is the common default for a reason: long enough to hold a paragraph with its context, short enough that the embedding is still about one thing.
• Match the size to the unit of meaning in your corpus. FAQ entries and changelog lines want 150 to 300 tokens; legal clauses, API reference pages and research papers want 800 or more. Look at the min and max in the stats strip: a wide spread means the strategy is fighting the document structure.
• Use overlap to protect sentences, not to pad. The job of overlap is to keep a sentence that straddles a boundary whole in at least one chunk. One or two sentences' worth, 40 to 100 tokens, does that. Beyond that you are paying to embed the same text twice for little gain.
• Prefer a structure-aware strategy over a bigger overlap. If the mid-sentence flag fires on many chunks, switching from fixed to recursive fixes the cause; raising the overlap only hides it at a cost that the Duplicated tokens number shows you.
• Go smaller for precise lookups, larger for synthesis. Questions with one factual answer retrieve best from small chunks; questions that need the model to summarise or compare do better with fewer, larger chunks. Many pipelines index both and let the query type decide.

Whatever you pick, re-run the visualizer when the corpus changes. Chunking settings tuned on one set of documents degrade quietly on the next.

Reading the flags and the stats strip

Each card in the chunk list shows the chunk's token and character count and the first eighty characters, and flags three kinds of trouble. Ends mid-sentence means the chunk does not end with terminal punctuation or a closing code fence, so a sentence was cut. Fragment means the chunk is under a fifth of the target size, which usually happens at the end of a document or right after an oversized block, and a fragment embeds as noise. Orphaned heading means the chunk's last line is a Markdown heading whose body went to the next chunk.

The stats strip puts the number most people have never seen in the middle: Duplicated tokens, the text that overlap repeats across neighbouring chunks. Every one of those tokens is embedded twice and stored twice, and the percentage next to it is the surcharge on top of embedding the document once. The last tile, Embedding tokens total, is roughly the document plus that surcharge (each chunk is counted on its own, so a boundary can move a token or two), the number to multiply by your embedding price. In character mode and with the Approximate tokenizer these are estimates and are marked with ≈; pick o200k or cl100k for exact counts.

Hover a card to highlight its text in the document, or hover the document to find its card. Striped regions are text shared by two chunks. Click a card to scroll the document to it.

Taking the chunks into your pipeline

Once the boundaries look right, Download JSON gives you an array of { index, text, tokens, chars } records and Download JSONL gives the same records one per line, the shape batch embedding jobs and most vector store loaders accept directly. The text in each record is exactly the highlighted span, overlap included, so what you export is what you saw.

{"index":0,"text":"# Chunking a knowledge base...","tokens":498,"chars":2102}
{"index":1,"text":"## Why size matters...","tokens":487,"chars":2044}

import json
chunks = [json.loads(line) for line in open("chunks.jsonl")]
vectors = embed([c["text"] for c in chunks])

Settings travel in the URL (?strategy=recursive&size=500&overlap=50), so Copy link hands a teammate the exact configuration without the document. To estimate what embedding or prompting the chunks will cost across models, paste a chunk into the Token Counter & LLM Cost tool, and to reshape the export for a batch API use the JSON ⇌ JSONL Converter. If your source is a web page, run it through HTML to Markdown first: the recursive strategy splits far better on Markdown headings and paragraphs than on raw HTML.