4 open-source tools that fix RAG before the model ever runs

Dear Reader…

Where RAG Is Won or Lost

Most RAG systems are capped long before the model is the bottleneck. The answers come back nearly right, the demo lands, and then in production the system quietly misses the one passage that mattered, and everyone blames the model. That gap between nearly right and genuinely trustworthy is the most common, most expensive and most fixable problem in applied AI, and it is almost never the model's fault.

This is the first of a few issues on closing that gap, by building a pipeline you actually own out of open-source parts rather than renting a black box that hides where things break. It starts at the foundation, because that is where the ceiling on everything downstream is set.

Ask an engineer to improve a RAG system and most will reach for a better model. It is the wrong move. The ceiling on what your AI can answer is decided long before the model runs, in how raw web pages and messy PDFs become clean, searchable knowledge. Get that wrong and no frontier model will save you. Reaching for the model is the mechanic's instinct; fixing the foundation is the architect's, and this year it is the more valuable one. Here are the four open-source tools that own that foundation, and what each is really for.

Get it in cleanly: Crawl4AI and Marker

Ingestion has two enemies: the web fights back, and documents lie flat.

Crawl4AI handles the web. Ordinary scrapers choke on JavaScript-heavy and single-page sites, and get themselves blocked. Crawl4AI is an asynchronous, high-concurrency crawler that renders those pages, adapts its crawl rate, and works around anti-bot friction. The quiet win is its output: it hands you clean, LLM-ready Markdown rather than raw HTML you then have to strip and de-noise, which removes an entire downstream cleanup stage. It is Apache-2.0 licensed, so there are no strings attached. The one thing to watch is memory bloat on very deep crawls, so keep your crawls bounded.

Marker handles the documents, and this is where most pipelines quietly lose. Ordinary PDF parsers flatten everything into a single text stream, discarding headers, reading order and, worst of all, tables. A financial table read as a wall of loose numbers is not data, it is noise, and it drags down every search that touches it. Marker converts PDF, PowerPoint, Word and image files into clean Markdown or JSON using the Surya layout and OCR engine, working block by block so it preserves table structure, including cells that span pages, and keeps equations intact as LaTeX. In its balanced mode it runs at roughly a fifth of a second per page.

One governance flag your finance and legal colleagues will want to know about before you commit: Marker's code is GPL-3.0, and its model weights carry a modified RAIL-M licence that requires a commercial licence once your organisation passes roughly two million dollars in lifetime funding or revenue. It is free to start and not free to scale silently. Better to know that now than at procurement.

Split it well: Chonkie

Once the documents are clean, you have to cut them into passages to embed and search. This is the step teams are laziest about, and the step that most often wrecks recall. Chop text every 500 characters and you will routinely slice a sentence, or a definition, clean in half, and the vector you then store is a fragment of a thought rather than a whole one.

Chonkie is a lightweight library, a 21MB footprint that matters in containerised pipelines, built to do this properly and fast. Its Rust core chunks quickly enough to segment the whole of English Wikipedia in about a tenth of a second, but raw speed is not the point. The algorithms are. Late chunking, for example, runs the entire document through the embedding model first and only then draws the boundaries, so every chunk carries a memory of the whole document instead of standing there context-blind. Its semantic splitters use signal smoothing to place boundaries at genuine topic shifts rather than at every small wobble in similarity, which stops one idea being carved into confetti.

The honest caveat: those clever semantic strategies are only as fast as the embedding model underneath them, and that embedding step, not the chunking, is the real bottleneck. Pair Chonkie with a lightweight static embedding model, or batch the work asynchronously, or you will simply move the queue rather than clear it.

Store it so it can be found: Qdrant

Now the passages need a home that can hold billions of them and return the right handful in milliseconds. Qdrant is a vector database built for exactly that, with distributed sharding coordinated by the Raft consensus protocol, the unglamorous machinery that keeps replicas consistent and recovers automatically when a node falls over.

The feature that earns its place, though, is hybrid search. Pure semantic search is brilliant at meaning and hopeless at exact strings. Ask it for part number 4F-9920-X or a specific contract clause and dense vectors will cheerfully return something that means roughly the same and is completely wrong. Qdrant combines dense semantic vectors with sparse lexical matching (BM25) and scores them together, so a single query can be both semantically smart and literally exact. For any real enterprise corpus full of codes, SKUs and identifiers, that is the difference between a system people trust and one they quietly stop using.

Why the foundation is the whole game

Here is the fair objection, because it deserves answering. Surely a strong enough model papers over a rough foundation? It does not, and the reason is simple. A language model can only reason over what retrieval hands it. If the right passage was never cleanly extracted, was split down the middle, or was buried under a near-miss because the search could not match an exact ID, the model never sees it, and no amount of prompt cleverness conjures back information that was lost three steps upstream. Retrieval sets the ceiling. The model only decides how close to it you get.

That is why this is the architect's stage, and why it pays. The engineer who reaches for a bigger model is competing with everyone else who can call an API. The one who can design ingestion, chunking and retrieval so the right context arrives every time is doing the part that is genuinely hard, harder to automate, and increasingly the thing that gets rewarded.

Next week we move up the stack, from a foundation worth querying to the layer that turns it into answers you can trust: DSPy, Instructor and Outlines, and the real difference between a demo and a product.

For now, if you take one thing from this issue, take this. Before you spend another week tuning the model, go and look hard at what you are feeding it. That is where the system is won or lost.

That’s a wrap for this week

Happy Engineering Data Pro’s