Build a grounded knowledge assistant

Scenario

A user asks questions about internal policies. The system retrieves relevant passages, gives them to a model, cites sources, and qualifies unsupported answers.

Concepts used

1. retrieval
2. grounding
3. chunking
4. citations
5. evaluation

Step-by-step build

1. Choose five short, current documents and assign source titles.
2. Create questions with known answers and two unsupported questions.
3. Start with simple keyword search before adding embeddings.
4. Pass only the best passages and explicit answer rules to the model.
5. Require source labels with every factual answer.
6. Decline when evidence is absent or contradictory.
7. Evaluate retrieval and answer quality separately.

Expected result

Supported questions receive concise cited answers; unsupported questions produce a useful limitation instead of a guess.

Failure cases

1. Right document not retrieved
2. Outdated passage outranks current policy
3. Citation does not support the claim
4. Model answers from general knowledge

Improvements

1. Add metadata filters
2. Introduce hybrid semantic search
3. Track unsupported-question rate

Platform variants

1. Simple application search
2. Microsoft Foundry
3. UiPath Context Grounding
4. Code-first RAG