Worked Example: Evaluating an LLM for Production

Here is the question: 'Your team is considering using a large language model (LLM) for a customer-facing product feature. How would you evaluate whether the LLM is ready for production?' This is one of the most common technical questions in 2024-2025 AI PM interviews. It tests your understanding of LLM capabilities, evaluation methodology, and production deployment challenges.

This question is particularly good because there is no single right answer. The evaluation approach depends on the use case, the risk tolerance, and the organizational context. The interviewer is testing your ability to ask clarifying questions, structure an evaluation framework, and identify the key risks.

"Before I design an evaluation framework, I need to understand the use case. Let me ask a few clarifying questions. First, what is the LLM being used for? Generation, classification, summarization, or conversation? Second, what is the risk level? Is this a recommendation (low risk if wrong) or a financial/medical decision (high risk)? Third, are we using a foundation model API or a fine-tuned model? This affects what we can evaluate and control."

"For this answer, I will assume we are evaluating an LLM for a customer support chatbot that answers product questions. This is a medium-risk use case: a wrong answer is not dangerous but can erode customer trust and increase support ticket volume."

"I would structure the evaluation in three phases: offline evaluation, human evaluation, and online evaluation. Each phase has a gate: we only proceed to the next phase if the current phase meets our quality bar."

[Interviewer note: Strong opening. The clarifying questions are relevant and concise (not stalling). The use case assumption is reasonable and stated explicitly. The three-phase framework is exactly how production LLM evaluation works. This candidate has either done this before or studied it carefully.]

"Phase 1: Offline Evaluation. I would build a benchmark dataset of 500+ question-answer pairs from real customer support tickets, with answers verified by domain experts. I would evaluate the LLM on this dataset using multiple metrics. Factual accuracy: what percentage of answers are factually correct? I would measure this with a combination of automated fact-checking (comparing answers against our knowledge base) and human review for a sample. Relevance: does the answer address the question asked? Measured by ROUGE-L against reference answers and by a separate LLM-as-judge scoring relevance on a 1-5 scale. Harmful content: does the model generate anything inappropriate, offensive, or that contradicts our policies? Measured by running outputs through a content safety classifier. Hallucination rate: does the model state things with confidence that are not in our knowledge base? This is the most important metric for a support chatbot. I would measure it by comparing generated claims against our ground truth and flagging any claims not supported by the source material."

"The quality bar for Phase 1: 95%+ factual accuracy, less than 2% hallucination rate, zero harmful content in the benchmark set. If the model does not meet these thresholds, we go back to the ML team for improvements before proceeding."

"Phase 2: Human Evaluation. I would run a red-teaming exercise with 10 domain experts who try to break the model by asking adversarial, edge-case, and ambiguous questions. Separately, I would run a blind evaluation where 5 support agents rate 200 model responses on a 1-5 scale for helpfulness, accuracy, and tone. The quality bar: average helpfulness rating of 4.0+ and no response rated 1 on accuracy."

[Interviewer note: Excellent evaluation design. The offline metrics are well-chosen: factual accuracy, relevance, harmful content, and hallucination rate are the right metrics for a support chatbot. The quality thresholds are specific and reasonable. The human evaluation phase adds the subjective quality check that automated metrics miss. The red-teaming mention shows awareness of adversarial robustness. Score so far: 4.5/5.]

"Phase 3: Online Evaluation. If the model passes Phase 1 and 2, I would deploy it to a small percentage of real traffic (5%) with a human-in-the-loop setup: the model generates the response, but a human agent reviews and approves it before it is sent to the customer. During this phase, I would measure: approval rate (what percentage of model responses agents send without modification), escalation rate (how often agents override the model and write their own response), customer satisfaction (CSAT score for AI-assisted interactions vs. fully human interactions), and resolution rate (are customers' issues actually resolved, or do they come back?)."

"The quality bar for removing the human-in-the-loop: approval rate above 90%, CSAT score within 5% of fully human interactions, and escalation rate below 10%. Once these thresholds are met, I would gradually expand to 25%, then 50%, then 100% of traffic, monitoring these metrics at each step."

"Post-launch monitoring: I would set up automated monitoring for response quality degradation over time (model drift), latency (P50, P95, P99), cost per interaction, and new topic detection (questions the model has not been evaluated on). If any metric degrades past a threshold, the system automatically routes those queries to human agents while we investigate."

[Interviewer note: The phased rollout with human-in-the-loop is exactly the right approach for a medium-risk customer-facing application. The candidate defined clear thresholds for each phase gate, included both model metrics and business metrics, and described a monitoring plan for post-launch. The automatic fallback to human agents on degradation is a production-mature design. This is a 5/5 on technical evaluation methodology.]

Overall score: 4.5/5 (Strong Hire). This answer demonstrates a structured, production-aware approach to LLM evaluation. The three-phase framework (offline, human, online) with clear quality gates is industry best practice. The metrics are specific and appropriate for the use case. The phased rollout and monitoring plan show operational maturity.

To reach a 5/5: The candidate could have discussed cost considerations (LLM inference costs can be significant for high-volume support), latency requirements (customers expect fast responses), and the specific challenges of evaluating LLMs compared to traditional ML models (non-deterministic outputs, prompt sensitivity, context window limitations). They could also have mentioned how they would handle the model's knowledge cutoff date and keeping the knowledge base up to date.