Asha Venkat runs research at Perigee AI, where her team's main output is not papers but decisions: which model behaviours matter, how to measure them, and when a number is good enough to ship against.
She built Perigee's evaluation practice after watching three launches judged on benchmarks that had almost nothing to do with what customers were doing. The replacement is unglamorous — a few hundred real tasks, graded by people who understand the domain.
Her team publishes its evaluation set internally so anyone can argue with it. The arguments, she says, are the point.
In Conversation
- What makes an evaluation useful?
- That someone could lose an argument to it. If no result would change what we ship, it is decoration.
- Papers or products?
- Papers tell you what is possible. Products tell you what is reliable. We need the second one to keep our promises.
- Biggest misconception about research teams?
- That we are upstream. We are in the room the whole time, or the work does not land.



