Technical Questions Scoring Rubric

Technical AI/ML questions are scored on four dimensions: Conceptual Understanding, Applied Reasoning, Production Awareness, and Communication. Unlike product sense questions, technical questions weight depth over breadth. A shallow answer that touches on many topics scores lower than a deep answer that thoroughly covers fewer topics.

The scoring scales are calibrated to level. A response that earns a 4 for an APM candidate might earn a 3 for a Senior PM candidate, because the bar for applied reasoning and production awareness increases with seniority. Interviewers are told to score against the level the candidate is interviewing for, not against an absolute standard.

This measures whether you understand the AI/ML concepts relevant to the question. It is the foundational dimension: if you do not understand the concepts, you cannot reason about them.

Score 1: Significant misconceptions about core concepts. Confuses classification with regression, does not understand what precision and recall measure. Score 2: Correct but surface-level understanding. Can define terms but cannot explain why they matter for product decisions. Score 3: Solid conceptual understanding. Can explain concepts in context, understands the relationships between concepts (e.g., precision-recall tradeoff). Score 4: Deep understanding. Can discuss nuances such as why accuracy is misleading for imbalanced datasets, when F1 is a better metric than accuracy, or how attention mechanisms enable different capabilities than recurrent architectures. Score 5: Expert-level understanding demonstrated through specific examples and edge cases. References real systems, published research, or industry benchmarks to support points.

Key phrases that signal strength: 'This depends on the distribution of the data.' 'In practice, the bottleneck is usually...' 'The theoretical advantage of X does not hold when...' Key phrases that signal weakness: 'AI would solve this.' 'We just need to train a model.' 'The model would learn to do X' (without explaining how or what data it needs).

This measures whether you can apply technical knowledge to make product decisions. This is the dimension that separates AI PMs from ML engineers: you are applying technical knowledge toward product outcomes, not solving technical problems for their own sake.

Score 1: Cannot connect technical concepts to product decisions. Score 2: Makes connections but they are generic or incorrect. 'We should use a larger model because larger models are better.' Score 3: Makes correct tradeoff decisions. 'I would choose approach A over B because our latency constraint of 100ms rules out B, even though B has higher offline accuracy.' Score 4: Makes nuanced tradeoff decisions that account for multiple factors (accuracy, latency, cost, user experience, organizational capability). Proposes creative solutions to technical constraints. Score 5: Demonstrates exceptional applied reasoning. Identifies non-obvious tradeoffs, proposes technical approaches that are both novel and practical, and shows deep understanding of how technical choices cascade into product outcomes.

The most common failure on this dimension is what interviewers call 'textbook answers': the candidate recites correct information but does not apply it to the specific scenario. If asked about A/B testing an AI feature and you describe a generic A/B test without addressing the AI-specific challenges (non-determinism, cold start, network effects), you will score a 2 even if your A/B testing knowledge is technically correct.

Dimension 3: Production Awareness (Weight: 25%). This measures whether you understand what it takes to deploy AI in production. Score 3 requires mentioning deployment considerations (latency, monitoring). Score 4 requires discussing data pipelines, model refresh cadence, fallback behavior, and monitoring for drift. Score 5 requires a production plan that addresses scalability, cost, incident response, and model governance.

Production awareness is the dimension where candidates with real AI shipping experience have the biggest advantage. If you have not shipped an AI feature, you can close this gap by studying public postmortem analyses and system design blog posts from companies like Uber, Airbnb, Netflix, and Google. Focus on: what went wrong in production that was not anticipated in development.

Dimension 4: Communication (Weight: 15%). Same principles as product sense: structure your answer, make tradeoffs explicit, and ensure the interviewer can follow your reasoning. For technical questions specifically, calibrate your technical depth to the audience. If the interviewer is an engineering manager, you can go deeper. If they are a product leader, stay at the applied level.

The overall pass criteria: average score of 3.5+ with no dimension below 3. The most common reject pattern on technical questions is scoring 4+ on Conceptual Understanding but 2 on Applied Reasoning. This profile says 'this person has studied the concepts but has not applied them.' The fix is to practice answering questions by making decisions, not just explaining concepts.