Building a trustworthy AI knowledge assistant
Build an AI knowledge assistant with curated sources, permission-aware retrieval, verifiable citations, and tests for unsupported or outdated answers.
A practical guide from Shivaantha · General guidance, not a report of client results.
Retrieval is a foundation, not a guarantee.
A retrieval-augmented system searches a collection of material and gives relevant passages to a language model. This can ground an answer in your organization’s information, but the system can still choose the wrong passage or draw an unsupported conclusion.
Evaluate retrieval and answer generation separately. If the correct evidence never reaches the model, changing the final prompt may not fix the problem. If the right evidence is present but the answer misrepresents it, the generation step needs attention.
Make the knowledge collection worth searching.
An index does not resolve contradictions between two policies or decide which abandoned document is authoritative. Give important sources an owner, a review process, and a clear status. Remove or clearly distinguish obsolete material.
Think about document updates and deletion before rollout. An answer may continue using removed information if the index and its caches are not refreshed. The lifecycle of a source should be connected to the lifecycle of its searchable representation.
Apply permissions before information is exposed.
A knowledge assistant should not use documents a requester is not allowed to read. Enforce access during retrieval and check that generated answers, source previews, and conversation history preserve those boundaries.
Test with users who have different roles and with permission changes after indexing. Do not rely on a model instruction to hide confidential information after it has already been retrieved into the model’s context.
Test questions the system should not answer.
A useful evaluation set includes straightforward questions, ambiguous questions, outdated facts, conflicting sources, and questions with no answer in the collection. For each case, identify what evidence would support a good response.
Check that citations actually support the claim beside them. A link to a real document can still be misleading if the answer goes beyond its contents. For unsupported questions, a clear statement of uncertainty and a route to a person are often the best result.
- Check source relevance and answer correctness independently.
- Test permission boundaries and deleted documents.
- Include malicious instructions embedded in source content.
- Review whether citations support each important claim.
- Make “not enough information” an acceptable outcome.