Why most “chat with your docs” projects lose trust
The first version of a company chatbot is easy to build and easy to be impressed by. Then someone asks about the parental-leave policy, gets a confident answer taken from a 2019 draft, and nobody uses the tool again.
The engineering that makes RAG trustworthy is unglamorous:
- how documents are split and cleaned
- whether search finds the right passage, not just a similar one
- how outdated versions are handled
- what the assistant does when it doesn’t know
That’s where most of a pilot’s time goes.
What’s under the hood
Ingestion that respects structure. Contracts, policies and manuals have sections, clauses and tables. I chunk along that structure and keep metadata like title, version, owner, date and permissions, so answers can be filtered and cited properly. Where documents are scanned PDFs, the same extraction pipeline used in document AI projects turns them into usable text first.
Hybrid retrieval and reranking. Semantic search finds related ideas. Keyword search finds exact terms like policy numbers, product codes or names. Combining both, then reranking the results, is the single biggest quality improvement in most projects.
Grounded answers with citations. Each answer links to the exact passage it came from. Readers can check the source in one click, which is what makes people trust the tool in professional settings.
Permissions. The assistant only retrieves documents the asking user is allowed to see, mirroring your existing access rules.
Evaluation as a feature. Every assistant ships with a test set of real questions and a repeatable scoring run. When you add documents, change models or tweak prompts, you re-run it and know whether quality went up or down.
What “accuracy” means in an evaluation report
“It seems to work” is not a measurement. Each pilot’s evaluation report scores the assistant on four things:
- Answer correctness: does the answer match what an expert on your team would say?
- Citation accuracy: does the cited passage actually support the answer?
- Retrieval recall: for each test question, did search surface the document that contains the answer?
- Refusal behaviour: when the answer isn’t in your documents, does the assistant say so instead of improvising?
These numbers show where to improve. Low retrieval recall usually means chunking or search needs work. Correct retrieval with wrong answers points to prompting or model choice. Without them, teams end up tweaking prompts at random.
Keeping it current
Documents change. Policies get revised, prices update, procedures are replaced. The sync pipeline watches your sources and re-indexes changed files, so the assistant answers from the current version. Superseded documents can be excluded or down-ranked automatically, which matters far more in practice than which model you use.
Where RAG assistants pay off
- Professional firms: precedents, templates, procedures and past advice, searchable in seconds.
- Operations and support: SOPs, product manuals and troubleshooting guides answered consistently for every agent.
- Onboarding: new hires ask the assistant instead of interrupting senior staff, and every answer comes with a source.
- Products: an in-app assistant for your customers, grounded in your documentation. It often pairs well with an MCP server, so AI agents can also act in your product, not just read about it.
You don’t need perfectly organised documents to start. Most knowledge bases are messy: duplicates, outdated drafts, half-finished wiki pages. Part of every pilot is a short report on content gaps and conflicts the assistant uncovered. That list is often valuable in its own right.
Industry research puts production RAG systems at roughly $8,000–25,000 for a solid first version. Larger, more integrated deployments cost more. A fixed-scope pilot is the lowest-risk way to find out what accuracy you can get on your own documents before committing further.