Worked Example: Decisions with Incomplete Data

Here is the question: 'Describe a situation where you had to make an important product decision with incomplete data. How did you approach it, and what was the outcome?' This question is especially relevant for AI PM roles because incomplete data is the normal operating condition in AI product development. You rarely have a perfect evaluation dataset. You rarely know how the model will perform on the long tail of real-world inputs. You rarely have enough data to be certain that your A/B test result will hold at scale.

Interviewers are not looking for a story where you waited until you had complete data. They are looking for a story where you identified what information was critical vs. what was nice-to-have, made a clear decision with the information you had, and set up a mechanism to correct course as new data came in.

"Situation: I was PM for a fintech product that helped small businesses manage cash flow. We wanted to add a feature that predicted upcoming cash shortfalls 2 weeks in advance, so business owners could take action (cut expenses, draw on a credit line, follow up on late invoices). The ML team built a model that predicted cash shortfalls with 73% precision and 65% recall on our test set."

"Task: I had to decide whether to launch the feature with these performance numbers. The data was incomplete in three ways. First, our test set was only 2,000 businesses over 6 months. We were not confident it represented the full diversity of our customer base (restaurants vs. consulting firms vs. retail). Second, we did not know the cost of a false positive (predicting a shortfall that does not happen) vs. a false negative (missing a shortfall). Was a false positive merely annoying, or would it cause panic and bad financial decisions? Third, we had no comparable product in the market, so we had no benchmark for what accuracy level users would accept."

[Interviewer note: Good problem setup. The three types of incomplete data (test set representativeness, cost of errors, user acceptance threshold) are clearly articulated and realistic. This shows the candidate understands the specific ways that data is incomplete in AI product development.]

"Action: I built a lightweight decision framework. I categorized the missing information into two buckets: 'must know before launch' and 'can learn post-launch.' The cost of errors fell into 'must know before launch' because launching a financial prediction feature that causes users to make bad decisions is a serious trust and regulatory risk."

"To fill this gap quickly, I ran a 2-week user research sprint. I showed 20 business owners mock notifications: 'Based on your recent invoicing patterns, you may experience a cash shortfall of approximately $8,000 in the next 14 days.' I then showed them: 'This prediction was wrong, your cash flow was fine.' I asked them how they would have reacted in each case. The results were clear: false positives were mildly annoying but not harmful, because business owners would verify the prediction against their own records before acting. False negatives were much worse, because a missed shortfall could mean bounced payments or missed payroll."

"This told me that recall mattered more than precision for this use case. A 65% recall meant we would miss 35% of real shortfalls. That was not acceptable. I went back to the ML team with a specific request: tune the model to increase recall to 80%+, even if precision dropped. They adjusted the threshold and got to 81% recall with 58% precision. This meant more false alarms, but the user research told us false alarms were acceptable."

"For the other gaps (test set representativeness and user acceptance), I put them in the 'learn post-launch' bucket. We launched to 500 users as a beta with clear language: 'Cash flow predictions (beta). These predictions improve as we learn your business patterns.' We monitored precision and recall by business type weekly and set a kill switch: if precision dropped below 40% for any segment, we would pause the feature for that segment."

[Interviewer note: The decision framework (must-know vs. can-learn-post-launch) is a practical tool that shows mature judgment. The user research sprint to fill the critical gap is efficient and appropriate. The precision-recall threshold adjustment based on user research shows real collaboration between product thinking and technical execution. The beta launch with monitoring and a kill switch is the right production strategy for incomplete-data situations.]

"Result: The beta launched successfully. After 3 months, the data showed that the model's accuracy varied significantly by business type. For businesses with regular invoicing (consulting firms, agencies), recall was 87%. For businesses with irregular cash flows (restaurants, retail), recall was only 64%. We used this data to segment the feature: showing predictions with higher confidence for regular-cash-flow businesses and showing them as 'possible trends' with lower confidence for irregular businesses."

"The feature became one of our most valued features with an NPS of 72, and it was a key driver in reducing churn by 8% among businesses that used it."

"AI Insight: In AI product development, waiting for complete data means never shipping. The skill is triage: determine which information gaps are critical (must fill before launch) vs. which you can learn from production data. For AI features specifically, the cost of different error types (false positives vs. false negatives) is the single most important thing to understand before launch, because it directly affects your model threshold and your user experience design. If you do not know the cost of errors, you do not know whether your model is good enough."

[Interviewer note: The AI Insight about triaging information gaps and understanding error costs is directly applicable to AI PM work. The result is quantified and shows sustained impact. Score: 4.5/5.]