RAG Text Chunker

Paste a document and see exactly where each chunk starts and ends, how much text overlaps and roughly how many tokens it will cost. Chunking and estimates run in the browser; nothing is uploaded or embedded.

Runs locally in your browser

View category: AI & LLM (7)

Source document

Chunk strategy

Chunk size (units)

Overlap (units)

Tune chunk boundaries before creating embeddings

Chunk size and overlap affect retrieval recall, context duplication and cost. This preview uses transparent heuristics; evaluate the resulting chunks against representative questions before production use.

How to use “RAG Text Chunker”

The chunker splits a document into deterministic retrieval units: fixed-size character windows, whitespace words, sentences, or Markdown heading sections, with an optional overlap carried from one chunk to the next.

Everything runs in the page. Chunk boundaries, character and token estimates and the JSONL export all come from the pasted text; the tool never calls an embedding model or a remote service.

Paste the document into the source pane and pick a strategy: characters, words, sentences or Markdown sections.

Set the chunk size in units of that strategy and the overlap in the same units; the overlap must stay smaller than the size.

Press Split document. The pipeline lists every chunk in order and the details card shows the character range and token estimate of the selected chunk.

Press Copy JSONL to export one JSON object per chunk with its id, text, range and estimated tokens.

Background and accuracy

What each strategy splits

Characters are Unicode grapheme clusters, so emoji flags, ZWJ families and combining marks are never cut in half. Words are whitespace-separated tokens; every CJK character counts as one word, so Chinese and Japanese text splits inside paragraphs that contain no spaces.

Sentences end at . ! ? or their full-width equivalents, and lines without sentence punctuation are kept as their own units instead of being dropped. Markdown sections start at ATX headings (# to ######); headings inside fenced code blocks are skipped, so a shell comment in a ``` block is not treated as a section boundary.

Boundaries, overlap and estimates

Each chunk is trimmed to its first and last non-whitespace character, and the reported range matches the text exactly. With overlap N, every chunk after the first repeats the last N units of the previous chunk, which protects facts that sit near a boundary.

The token estimate is a heuristic, not a tokenizer: CJK characters count about 1.05 tokens each and other scripts about one token per four characters. Use it to compare strategies, then measure real token counts with the tokenizer of the model you deploy.

Working with the JSONL export

Copy JSONL writes one object per chunk — id, text, start, end and estimated_tokens — in reading order. The ranges refer to the pasted source, so overlapping chunks can be mapped back to the original document.

The export is ready for an embedding pipeline, but this page does not create embeddings and does not upload text. Keep the JSONL local, or send it only to the service you intend to use.