How to Chunk Documents for RAG: 3 Methods Compared

Your retriever can only return what a single chunk contains, so the way you split documents sets the ceiling on every answer your RAG system will ever give. The short version: start with a recursive character splitter at roughly 512 tokens with about 15% overlap, measure retrieval on a real question set, then reach for structure-aware or semantic splitting only where the data demands it. Chunking is the one preprocessing decision you make before any model runs, and it is the cheapest thing to get wrong.

Key Takeaways

Short answer: use a recursive character splitter at about 512 tokens with 10%-20% overlap as your baseline, then tune against an eval set.
Measure chunk length in tokens, not characters, so every chunk fits inside your embedding model's context window.
Overlap keeps a sentence that straddles a boundary retrievable from either neighbor; 10%-20% is a sensible start.
Structure-aware splitting on Markdown, HTML, or code beats blind character splitting whenever your documents have clear sections.
Semantic chunking can raise coherence but spends an embedding call per sentence and does not reliably win independent benchmarks.
You cannot judge a chunking change by eye; prove it with retrieval metrics and change one variable at a time.

What You Need Before You Start

This guide uses Python and LangChain's text splitters, which are the most widely used and are easy to swap for the LlamaIndex equivalents.

Python 3.9 or newer.
The splitter and tokenizer packages: pip install langchain-text-splitters tiktoken.
Your source content already extracted to text. PDF, DOCX, and HTML extraction happens before chunking, and messy extraction ruins good chunking.
A small eval set: 20 to 50 real questions with the passage or answer you expect back. Without it you are guessing.
Optional, for semantic chunking: langchain-experimental and an embeddings provider.

Chunk Your Documents in Six Steps

Follow these in order. The first four get a working baseline you can index and query today; steps five and six handle structured data and hand off to your vector store.

Install the splitter and a tokenizer.

pip install langchain-text-splitters tiktoken

Load and clean your raw text. Strip navigation, headers, footers, and repeated boilerplate. Junk that survives extraction becomes junk chunks that pollute retrieval.
Split with a recursive character splitter, measured in tokens. This tries natural boundaries first (paragraphs, then lines, then sentences, then words) and only cuts mid-word as a last resort. Using from_tiktoken_encoder makes chunk_size count tokens, which is what your embedding model actually limits.
```
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=512,
    chunk_overlap=64,
)

chunks = splitter.split_text(raw_text)
print(len(chunks), "chunks")
```
The default separators are ["\n\n", "\n", " ", ""], tried in that order. If you have LangChain Document objects with metadata, call splitter.split_documents(docs) instead so source metadata rides along with each chunk.

Inspect boundaries before you index anything. Look at the shortest and longest chunks and read a few by hand. Tiny fragments and chunks that end mid-thought are signals to adjust size or overlap.

lengths = [len(c) for c in chunks]
print("count:", len(chunks))
print("min/max chars:", min(lengths), max(lengths))
print(chunks[0][-200:])   # tail of the first chunk
print(chunks[1][:200])    # head of the second, to see the overlap

For structured documents, split on structure first. Markdown, HTML, and source code carry their own boundaries. Split on those, then recursively split any oversized section so nothing exceeds your token budget.

from langchain_text_splitters import MarkdownHeaderTextSplitter

headers = [("#", "h1"), ("##", "h2"), ("###", "h3")]
md = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
sections = md.split_text(markdown_text)   # keeps heading path in metadata

chunks = splitter.split_documents(sections)   # re-split long sections by tokens

Embed the chunks and load them into your vector store. Chunking feeds the same embed-and-index step every retriever uses. If you are new to that half, wire it up with a local RAG pipeline or store vectors in Postgres with pgvector.

Which Chunking Strategy Should You Use?

Four strategies cover almost every real corpus. Recursive is the default workhorse; the others earn their place only against specific document shapes.

Strategy	How it splits	Best for	Cost	Watch out for
Fixed-size	Every N tokens, ignoring content	Uniform text, quick prototypes	Cheapest	Cuts sentences and ideas in half
Recursive character	Natural boundaries first, falling back to smaller ones	Almost everything; your baseline	Cheap	Blind to headings and tables
Structure-aware	On document markup: headings, HTML tags, code blocks	Docs, wikis, API references, code	Cheap	Needs clean, consistent markup
Semantic	On embedding shifts between adjacent sentences	Long prose with drifting topics	An embedding call per sentence	Variable chunk sizes; slower; not always better

How Big Should a Chunk Be, and How Much Overlap?

Size trades recall against precision. Larger chunks carry more context but dilute the signal an embedding represents, so retrieval pulls back loosely related material. Smaller chunks retrieve sharply but can arrive at the model missing the context needed to answer.

A 512-token chunk with 64 tokens of overlap (about 12%) is a defensible default for mixed prose. Drop toward 256 tokens when your questions are pointed and factual; climb toward 800 to 1,000 when answers need surrounding narrative. Overlap of 10%-20% is the standard band: enough to rescue a thought split across a boundary, not so much that you bloat the index with duplicated text.

Pick token-based sizing, not character-based. Embedding and generation limits are expressed in tokens, and a character count that looks safe can blow past a model's window on dense text.

When Is Semantic Chunking Worth the Cost?

Semantic chunking groups adjacent sentences by embedding similarity and cuts where the topic shifts, so chunk boundaries follow meaning instead of punctuation. It shines on long, unstructured prose where a single heading covers several distinct ideas.

It also embeds every sentence to find those breakpoints, which costs real money and time on a large corpus, and independent 2026 benchmarks have repeatedly shown recursive 512-token splitting matching or beating it on end-to-end accuracy. Treat it as an experiment you validate, not a default.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

chunker = SemanticChunker(
    OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",   # or standard_deviation, interquartile, gradient
)
docs = chunker.create_documents([raw_text])

The breakpoint_threshold_type controls how aggressively it cuts. percentile is the usual starting point; gradient tends to help on dense technical text where topic shifts are gradual.

Troubleshooting Common Chunking Problems

Chunks exceed the embedding model's token limit. You are measuring characters. Switch to RecursiveCharacterTextSplitter.from_tiktoken_encoder(...) so chunk_size counts tokens.
Answers miss facts that clearly exist in the source. The fact likely sits on a chunk boundary. Raise chunk_overlap or split on structure so related sentences stay together.
Retrieval returns vaguely related passages. Your chunks are too large and their embeddings are muddy. Reduce chunk_size and re-measure precision.
Tables and code come back mangled. Character splitting tore them apart. Use a structure-aware splitter and never split a code block or table row mid-unit.
SemanticChunker import fails. It lives in langchain_experimental.text_splitter, not the core package. Install langchain-experimental.
Semantic chunking is slow and expensive. Expected: it embeds every sentence. Batch the embeddings, or fall back to recursive splitting for bulk ingestion and reserve semantic for a high-value subset.

What to Do Next

Chunking is one input to retrieval quality, so tune it inside a loop that measures the whole pipeline.

Build the eval loop first, then A/B your chunk sizes against it. See how to run evals on your LLM app with Promptfoo.
Once chunks are sound, fix ranking with a reranker on your RAG pipeline before you touch chunking again.
Read the primary splitter docs: LangChain's RecursiveCharacterTextSplitter reference and its splitting by token guide.