Agent-as-a-Judge: Evaluate Agents with Agents // paper
Contemporary evaluation techniques are inadequate for agentic systems. These approaches either focus exclusively on final outcomes — ignoring the step-by-step nature of agentic systems, or require excessive manual labour. To address this, we introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We apply the Agent-as-a-Judge to the task of code generation. To overcome issues with existing benchmarks and provide a proof-of-concept testbed for Agent-as-a-Judge, we present DevAI, a new benchmark of 55 realistic automated AI development tasks. It includes rich manual annotations, like a total of 365 hierarchical user requirements. We benchmark three of the popular agentic systems using Agent-as-a-Judge and find it dramatically outperforms LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether, we believe that Agent-as-a-Judge marks a concrete step forward for modern agentic systems — by providing rich and reliable reward signals necessary for dynamic and scalable self-improvement
The field of artificial intelligence is rapidly evolving, and with it, the need for more effective evaluation methods has become increasingly important. Traditional evaluation techniques often focus exclusively on final outcomes, ignoring the step-by-step nature of agentic systems. This can lead to incomplete assessments and a lack of understanding of the complex processes involved. To address this issue, researchers have introduced the Agent-as-a-Judge framework, which uses agentic systems to evaluate other agentic systems.
The Agent-as-a-Judge framework is an extension of the LLM-as-a-Judge framework, which uses large language models to evaluate other language models. By incorporating agentic features, Agent-as-a-Judge provides rich intermediate feedback throughout the entire process, enabling a more comprehensive assessment of the system. This approach has been applied to the task of code generation, where agentic systems are used to evaluate the code-generating abilities of other agentic systems.
To facilitate the evaluation process, researchers have also introduced DevAI, a new benchmark dataset consisting of 55 realistic AI code generation tasks. Each task includes a user query, requirements, and preferences, providing a comprehensive understanding of the development process. DevAI was designed to reflect practical software scenarios, emphasizing the development process over final outcomes. By using DevAI with Agent-as-a-Judge, researchers can gain a more detailed understanding of the strengths and weaknesses of code-generating agentic systems.
The results of the evaluation process are impressive, with Agent-as-a-Judge outperforming existing methods and performing similarly to an ensemble of expert human evaluators. The framework’s ability to provide intermediate feedback and its emphasis on the development process make it a valuable tool for improving the performance of agentic systems. By leveraging Agent-as-a-Judge, researchers can create more sophisticated and reliable agentic systems, leading to breakthroughs in various fields, including code generation.
The introduction of Agent-as-a-Judge and DevAI marks a significant step forward in the field of artificial intelligence. By providing a more comprehensive evaluation framework, researchers can now develop more advanced agentic systems, leading to increased efficiency and productivity. The potential applications of this technology are vast, ranging from improved code generation to enhanced decision-making capabilities. As AI continues to evolve, the importance of effective evaluation methods will only continue to grow, making Agent-as-a-Judge a vital tool for researchers and developers in the field.
