Evaluating RAG & Catching Hallucinations
Measure a RAG system in two halves: retrieval (recall@k, MRR, nDCG) and answers (faithfulness, relevance), with a golden set, a calibrated LLM judge, CRAG-style filtering, regression gates and online signals.
Overview
Why this topic
Without measurement, every RAG change is an opinion. A new chunk size, a reranker, a prompt edit or a model upgrade can each help one kind of question and quietly break another. Evaluation turns those changes into numbers you can compare, and hallucination checks catch the answers that sound right and are not supported by any source.
The core idea: evaluate retrieval and generation separately, because they fail for different reasons and have different fixes.
Interview questions
- How would you set up evaluation for a new RAG system from scratch?
- Explain recall@k, MRR and nDCG with an example. When would you use each?
- How do you measure faithfulness, and what is the difference from correctness?
- You use an LLM as a judge. What can go wrong, and how do you make its scores trustworthy?
- How would you stop a RAG system from answering when retrieval returns irrelevant chunks?
- A prompt change shipped and users say answers got worse, yet your offline eval passed. What happened and what do you change?
Sign up free to read the full topic
More Agentic AI topics
- Context Engineering & Prompt Design
- Context Window Management for Persistent Agents
- Designing Tools a Model Calls Well
- Distillation & Small Models
- Embeddings & Vector Search
- Evaluating Agents: Task Suites, Trajectories & Regression Gates
- Fine-tuning LLMs: SFT, LoRA/QLoRA & Preference Tuning
- Guardrails & Prompt Injection Defense
- LLM Cost & Latency Engineering
- LLM Observability: Traces, Spans & OpenTelemetry GenAI
- Model Context Protocol (MCP) Servers
- Multi-Agent Systems with LangGraph
Start free on Cariara · Coding problems · System design studio