AI Development Services
RAG Chatbot Development: Building a Company Knowledge Assistant
A RAG (retrieval-augmented generation) chatbot answers questions by first searching your own documents and data, then letting a language model write an answer based only on what it found, ideally with sources. Done well, it gives employees or customers fast, verifiable answers; done poorly, it confidently cites the wrong policy. The difference lies almost entirely in data preparation, retrieval quality and permissions.
How a RAG knowledge assistant works
A RAG system has two halves: a retrieval layer that finds the most relevant pieces of your content, and a generation step where an LLM composes an answer from those pieces.
Your documents are split into chunks, converted into embeddings (numeric representations of meaning) and stored in a vector index, often alongside a classic keyword index. When a user asks a question, the system retrieves the best matching chunks, passes them to the model with instructions to answer only from that context, and returns the answer with references. Nothing is retrained; you can update knowledge by updating documents.
Good use cases and bad ones
RAG shines where knowledge exists in writing but is hard to find.
Typical wins are internal handbooks and HR policies, IT and product support, technical documentation for field staff, sales enablement content, contract and tender archives, and customer self-service on top of help-center articles. RAG is a poor fit when the answer requires calculations across live transactional data; there, tool calls into your systems or classic reporting work better, sometimes combined with RAG for the explanatory part.
Data preparation decides answer quality
Most bad answers are retrieval failures, not model failures. If the right paragraph is never retrieved, no model can answer correctly.
Before indexing, clean up duplicates and outdated versions, extract text properly from PDFs and scans, keep headings and table structure, and attach metadata such as document type, date, department and access level. Chunking strategy matters: chunks that are too small lose context, chunks that are too large dilute relevance. Hybrid search, combining semantic and keyword matching, and a re-ranking step usually improve results noticeably for company jargon, product codes and names.
- Remove outdated and duplicate documents first.
- Preserve document structure and tables.
- Store metadata for filtering and permissions.
- Use hybrid retrieval plus re-ranking.
- Re-index automatically when sources change.
Permissions: never let the bot leak what a user cannot see
A knowledge assistant must respect the same access rules as the source systems, or it becomes a data leak with a friendly interface.
Filter retrieval by the user's permissions before anything reaches the model, using group or role metadata synced from your identity provider and the source systems such as SharePoint, Confluence or a document management system. Do not rely on the model to withhold information it has already been given. This is the single most important design decision in enterprise RAG.
Evaluation and continuous improvement
Measure before launch and keep measuring afterwards.
Build a test set of real questions with known correct sources and answers, ideally collected from the people who will use the assistant. Track whether the right sources are retrieved, whether answers are faithful to them, and how often the assistant correctly says it does not know. After launch, collect thumbs-up and thumbs-down feedback, review unanswered questions weekly, and use them to fix content gaps. Often the assistant's biggest value is revealing which documentation is missing or contradictory.
Hosting, privacy and the build process
Choose hosting based on data sensitivity: EU-region cloud services under a data processing agreement work for many cases, while highly sensitive data may call for EU-hosted or self-hosted open-weight models.
A typical project runs through these phases.
- Use-case workshop: target users, top questions, sources, success metric.
- Data audit and connector setup for the chosen sources.
- Prototype with a test set and a small pilot group.
- Permissions, logging, privacy documentation and security review.
- Rollout with feedback loop and regular content reviews.
Build, buy or platform?
Off-the-shelf assistants built into office suites cover simple cases quickly. Custom development pays off when you need specific sources, strict permission handling, your own UI or integration into existing apps and workflows.
For enterprises and public sector bodies that want a compliant platform rather than a one-off project, our sister brand kalAIDoCreate is worth a look. For custom assistants embedded in your own products, neooutput's AI Code Engineers build and maintain RAG systems end to end.
Frequently asked questions
Does a RAG chatbot hallucinate?
It can, but much less when retrieval is good, the prompt instructs the model to answer only from the provided context, and sources are shown. Evaluation and an explicit I don't know behavior are essential.
Do we need to train our own model?
Almost never. RAG works with existing models and keeps knowledge in your documents, which makes updates instant and avoids costly training.
Which document sources can be connected?
Common sources include SharePoint, Confluence, Google Drive, file shares, ticket systems, wikis, databases and websites. Anything with an API or export can usually be connected.
How long until a first pilot?
With clean sources and a defined use case, a working pilot for a small group is often possible within a few weeks. Messy data and complex permissions take longer.
