I built Mabel assuming RAG was basically a solved problem. Pull chunks, stuff them in the prompt, let the model handle it. First version worked fine on clean PDFs, nearly shipped it. Then I tested it on a scanned textbook, footnotes everywhere, three columns, inline citations. It started making up authors that didn't exist.
Nobody really warns you about this part. Everyone talks about retrieval like that's where it breaks. But the failure that actually hurts users isn't bad retrieval. It's good retrieval that the model just ignores, or worse, takes and bends into something that sounds confident but is completely wrong.
The chunking trap
Every tutorial says chunk at 500 tokens, add some overlap, move on. Works fine on clean prose. The moment you have any real structure it falls apart. Tables get cut mid-row. Code blocks get separated from the comment explaining them. Footnotes end up orphaned with no context.
What actually helped was treating it like a parsing problem. Extract the structure first: headings, tables, code blocks as whole units. Then figure out where to cut. Chunks got longer and less uniform. I'd been told that was bad. Answers got way more grounded.
Multiple sources, multiple problems
Pulling from multiple sources isn't just doing retrieval twice. When someone uploads a doc and also wants me to check the web, those two things have completely different reliability. Their textbook is high trust. A random scraped blog post is not.
Web retrieval kept being brittle so I built SiteScan on the side, a small service that takes any URL and turns it into clean LLM-ready text in one call. Stripped most of the noise from raw scraping. Web context started behaving closer to a proper document.
Mixing both sources in the same prompt with no labels is how you get the model to just average them together and trust the wrong one. Making the source explicit in the context helped a lot. Something like [from your document, page 47] vs [from the web], then telling the model to prefer the doc when they conflict. Simple, but it worked.
What I'm still figuring out
The hardest thing right now is getting the model to know when to not answer. People ask stuff their documents don't cover, and "based on your document, X seems likely" is exactly the kind of hallucination that kills trust. Getting a model to say "this isn't in your document" is weirdly harder than getting it to answer well. The training pushes it to always be helpful, not to admit it doesn't know.
No clean solution yet. I've been running a separate verification pass that re-reads the retrieved chunks and just asks "does this actually support the claim?" and rejecting anything that fails. Slower, costs more tokens. Also the only thing I've tried that actually reduces hallucinations in prod.