RAG — Interview Perspective

The question

“Walk me through how you would design and evaluate a RAG system for enterprise knowledge.”

Senior and principal interviews rarely stop at the definition — they probe for production scars. What follows is how I structure the answer, the follow-ups to expect, and what the interviewer is actually evaluating.

My approach

I answer in three layers, in this order — the same order I would build:

1. Ingestion: sources → structure-aware chunking → embeddings → index with metadata. Emphasize that chunking respects source structure, because that decision outlives every model choice.
2. Retrieval: hybrid (vector + keyword) with metadata filtering and optional reranking; tiered precedence when sources conflict.
3. Generation: grounded, structured, cited output with explicit refusal when context is insufficient — then evaluation layered over all of it (retrieval metrics and generation metrics, separately).

Framing matters: I present RAG as an information-retrieval problem with a generation step, not as “an LLM feature”. That framing alone signals seniority.

Strong answer markers

• Retrieval evaluated in isolation (hit rate / MRR against labeled chunks) before touching prompts.
• Hybrid search justified by corpus content (identifiers), not by fashion.
• Precedence rules encoded structurally, not delegated to model judgment.
• Refusal quality treated as a first-class metric.
• A real production story: what regressed, how traces localized it, what was changed, what is now monitored.

Common mistakes I see

• Reciting the pipeline (chunk → embed → retrieve → generate) without a single trade-off.
• Claiming fine-tuning is a substitute for RAG on mutable enterprise knowledge.
• No separation between retrieval evaluation and generation evaluation.
• Hand-waving access control (“we filter in the prompt”).
• No story about a failure. Everyone has had one; not having looked is the red flag.

Follow-up questions to prepare for

• “Why not fine-tune instead?” — freshness, provenance, per-domain corpora, auditability.
• “How do you measure retrieval quality?” — hit rate, MRR, recall@k on a labeled golden set.
• “How do you handle conflicting sources?” — structural precedence, deterministic resolution.
• “How would you cut latency by 50%?” — reranking budget, caching strategy, smaller context with tighter construction, async where the UX allows.
• “How do you handle PII?” — enforcement inside retrieval, index lifecycle as a first-class data-governance problem.

What interviewers are evaluating

At senior/principal level, this question is not testing whether you know what an embedding is. It is testing:

• Systems thinking — do you reason in trade-offs (latency, cost, quality, governance)?
• Operational maturity — evaluation gates, observability, failure stories.
• Honesty about limits — do you know where RAG fails, and what you do about it?
• Experience density — whether the answers sound lived-in or read-in.

The strongest candidates say “I got this wrong once, here is what changed” — the weakest recite best practices they have never had to defend.