Demystifying evals for AI agents
Anthropic lays out its field-tested framework for evaluating AI agents: the vocabulary (tasks, trials, graders, transcripts, harnesses), the three grader types (code-based, model-based, human), capability vs. regression evals, and two key non-determinism metrics, pass@k (odds of at least one success in k attempts) and pass^k (odds all k attempts succeed), plus an 8-step roadmap for building an eval suite from scratch.
More from Anthropic
Effective context engineering for AI agents
Anthropic shares principles for building effective AI agents, focusing on context engineering — the practice of providing agents with the right information, tools, and instructions to produce reliable results. Covers prompt design, tool definitions, and managing long-running agent sessions.
How we built our multi-agent research system
Anthropic describes the engineering behind their multi-agent Research feature, where a planning agent decomposes complex queries and spawns parallel search agents. The post covers architectural principles, prompting strategies, and evaluation methods for reliable multi-agent systems.
