Markdown Is Step One. Chunking Is Where RAG Actually Breaks.

Most RAG tutorials spend their energy on the wrong step. They fuss over which vector database to use and which embedding model is a few points higher on some leaderboard, then split the documents with a one-liner that chops text every N characters. The database and the embedding model are rarely why retrieval disappoints. The chunking is. Converting your PDFs to clean Markdown is genuinely useful, but it's step one — the point where things are still going well. Chunking is the step where a pipeline quietly breaks, and it breaks in ways that don't show up until a user asks a question and gets a confidently wrong answer built from the wrong three paragraphs.

This post is about that step: why fixed-size splitting hurts, what a sane default actually looks like, why dirty text poisons your embeddings, and why the metadata attached to each chunk matters as much as the text inside it.

Why you still chunk in the age of huge context windows

A reasonable objection: models now take hundreds of thousands of tokens. Why split anything? Two reasons that don't go away no matter how big the context window gets.

First, embedding models have hard token limits, and they're small. The model that turns your text into the vectors your retrieval depends on typically caps out in the low thousands of tokens — nothing like the context window of the chat model downstream. You cannot embed a 90-page document as one vector even if you can prompt with it. Retrieval requires pieces.

Second, "lost in the middle" is real. Even when everything fits, a model's ability to use information degrades for content buried in the middle of a long input; recall is strongest at the beginning and end. Stuffing the entire corpus into context and hoping the model finds the relevant clause is a worse strategy than retrieving a handful of focused, well-formed chunks and handing over only those. Chunking isn't a workaround for small context windows. It's how you feed the model the right tokens instead of all the tokens.

Fixed-size splitting is the default, and it's the problem

The most common chunker splits on a raw count — every 1,000 characters, or every 500 tokens, hard stop, wherever that lands. It is trivial to implement, which is why it's everywhere, and it treats your document as an undifferentiated stream of characters. That's the flaw.

Documents are not streams. They have structure — sections, subsections, tables, definitions — and that structure carries meaning. A fixed-size cut is blind to all of it. It will sever a sentence mid-clause. It will split a table so its header lands in one chunk and its rows in the next, leaving both halves meaningless. It will fuse the last paragraph of a section about termination with the first paragraph of a section about governing law, producing a chunk that is topically incoherent and embeds to a vector that represents neither idea well. When you later query for termination terms, the relevant text is diluted by governing-law text it happened to be sitting next to, and your retrieval quality pays for it.

Header-aware chunking, and a default that works

The better approach uses the structure the document already gives you. Markdown makes this practical because headings are explicit: #, ##, ### are unambiguous boundaries a splitter can respect. Header-aware chunking breaks at those boundaries first and only falls back to size limits within a section. It tends to outperform fixed-size splitting for exactly the reason fixed-size fails — chunks line up with the document's own units of meaning, so each one is about a single thing.

Published guidance and practical experience converge on a working range rather than a magic number:

• Chunk size: roughly 600–1,000 tokens. Small enough to stay under embedding-model limits and keep each vector focused; large enough to carry a complete thought instead of an orphaned fragment. MarkPrep defaults to 800 tokens, adjustable from 100 to 2,000 for when your content or embedding model wants something different.
• Overlap: roughly 50–150 tokens. A sliding window that repeats a little text between adjacent chunks so an idea straddling a boundary survives in at least one piece intact. MarkPrep defaults to 100 tokens of overlap. Too little and you guillotine cross-boundary context; too much and you bloat your index with redundancy and pay to embed the same sentences repeatedly.

Treat these as starting points you validate against your own retrieval quality, not commandments. The important move is smaller: chunk by structure first, size second, and pick numbers in the sane range instead of whatever the tutorial pasted.

Approach	Boundary logic	Typical failure
Fixed-size	Every N tokens, blind	Split tables, severed sentences, mixed-topic chunks
Header-aware	At headings, size as fallback	Uneven chunk sizes (usually fine)

Dirty text poisons the vectors before you ever chunk it

Here is the failure that precedes chunking and quietly wrecks it: the Markdown itself is dirty. Raw PDF extraction drags along page furniture — the running header repeated on all 90 pages, the footer with the confidentiality notice, the page numbers, the "Page 12 of 90," the line that got hyphenated across a page break and never rejoined.

Feed that into a chunker and every consequence compounds. The repeated header shows up in every chunk, so every vector shares a slug of identical boilerplate that pulls unrelated chunks closer together in embedding space and blurs the distinctions retrieval depends on. Page numbers and furniture spend tokens — your token budget and your embedding budget — on text that carries zero semantic value. A word split as inter- / national across a line break embeds as two fragments instead of the concept "international." The chunk boundaries you carefully aligned to headings don't help if the text inside them is littered with noise.

So clean-up is not cosmetic; it is a precondition for chunking to work. MarkPrep runs a clean-up engine by default that strips repeated headers and footers, removes page numbers, de-hyphenates words broken across line breaks, collapses runs of blank lines, and normalizes tables. The point is to hand the chunker semantic content and nothing else, so the vectors represent what the document says rather than how it was printed.

Metadata is what turns a chunk into a retrievable unit

A chunk is not just its text. What you attach to it decides how useful it is once it's retrieved.

Heading path. Each chunk should carry the trail of headings above it — Master Services Agreement > Liability > Limitation of Liability. That path does real work. It tells you where a retrieved chunk came from, so you can cite it. It lets you filter or boost retrieval by section. And it gives the chunk context its own text may lack — a chunk that says "such liability shall not exceed the fees paid" is ambiguous alone and unambiguous under the heading path that locates it.

Page number. Keep the source page. When a model cites a chunk, you can point a human back to page 47 of the original PDF to verify it. In any domain where answers must be checkable — legal, medical, financial — traceability isn't a nice-to-have; it's the difference between a usable system and a liability.

With this metadata in place, retrieval stops being pure vector similarity and becomes something you can reason about: find chunks similar to the query, prefer the ones under the "Indemnification" heading, and show me their source pages. The metadata is what makes that query expressible.

Ship it as JSONL

Retrieval pipelines want structured, line-delimited records, not a wall of Markdown. JSONL — one JSON object per line — is the lingua franca: stream it, load it row by row, push it straight into an indexing job. Each line pairs the chunk text with its metadata:

{"text": "Such liability shall not exceed...", "headingPath": ["MSA", "Liability", "Limitation of Liability"], "page": 47, "tokens": 780}

MarkPrep exports exactly this — heading-path and page metadata per chunk — so the artifact that leaves the tool is the artifact your pipeline ingests. No glue script to reattach the structure that a bare .md file threw away, because it was never thrown away.

The order that actually matters

Put the steps in order and the priorities invert from how they're usually taught. Clean the text so the vectors mean something. Chunk on structure, not on a blind character count. Pick a size and overlap in the sane range — 600–1,000 tokens, 50–150 overlap — and tune against your own results. Attach heading path and page to every chunk. Export JSONL your pipeline can read directly. The vector database and the embedding model, the parts tutorials obsess over, are the parts most forgiving of getting slightly wrong.

MarkPrep does the conversion, then shows you the chunk plan before you commit to it — heading-aware boundaries, adjustable size and overlap, per-chunk metadata, JSONL out. See how your document actually splits: convert a file and open the chunk view, or start with the token counter to size the job. Clean Markdown was step one. Chunking is the step that decides whether any of it retrieves.