An agent that demos well can still be a liability in production. Here’s how to test one honestly.
An agent that demos well can still be a liability in production. Before you let one touch real work, put it through a short, honest evaluation.
Test it on your messy reality
Demos use clean inputs. Your work doesn’t. Give the agent a genuinely ambiguous task from your own backlog and watch where it guesses, stalls, or quietly does the wrong thing.
Judge recovery, not just success
The best agents fail gracefully — they ask, escalate, or stop instead of plowing ahead. How an agent behaves when it’s unsure tells you far more than a perfect happy-path run.
