pass at k and pass to the kth measure opposite qualities

pass@k measures the likelihood of at least one success (see: evals define what success means for an agent) in k attempts; it rises as k grows. pass^k measures the probability all k trials succeed; it falls as k grows. By k=10 they tell opposite stories: pass@k approaches 100% while pass^k falls toward zero. Use pass@k when one success matters; use pass^k when consistency is essential.

The practical implication is that you must choose your metric based on product requirements. A coding tool where one good solution suffices should optimize pass@1: you want the first attempt to work. A customer-facing agent where users expect reliable behavior every time should optimize pass^k: you want consistency across runs. Reporting only pass@k creates a false sense of capability, because it hides how often the agent fails on any given attempt.