The summariser that invents dates

A product case question from a product manager interview, set at Gistwell, a fictional enterprise genai / summarisation company.

The question

Skills this case tests

A starting metric tree

Wrong-date rate = summaries with a date error ÷ summaries; split by thread type and error type, measured by tickets and by a random audit

How to approach this case

Start by defining the failure. Is it a wrong day, a wrong owner, or a deadline that was never in the thread? Ask how many customers are affected and how serious it is, since people act on these deadlines. Customer tickets under-count errors, so plan a random audit as well. Then segment the errors by the kind of thread and the kind of mistake. List everything that changed at launch, including the model, the prompt and the steps that prepare the input, and design a test that proves or rules out each one. Then ask how the launch passed: what the eval measured and how it was built. A strong close fixes the issue for customers now, sets a launch bar for future model and prompt changes, and weighs the cost and latency of each option.

The full method for mixed diagnose-and-decide questions · A worked example answer

After the mock interview you get scores on five skills, what to fix with quotes from your answers, and the ideal path for this case.