Self-improving Multi-agent Systems // no humans needed, a joke
The benefits of multi-agent systems are internal competition. communication and shared knowledge
Multi-agent AI systems powered by large language models (LLMs) are increasingly applied to solve complex tasks. However, these systems often rely on fragile, manually designed prompts and heuristics, making optimization difficult. A key challenge in optimizing multi-agent systems is acquiring suitable training data for specialized agents. We introduce SIRIUS, a self-improving, reasoning-driven optimization framework for multi-agent systems. Central to our approach is the construction of an experience library: a repository of high-quality reasoning trajectories. The library is built by retaining reasoning steps that lead to successful outcomes, providing a robust training set for optimizing multi-agent system. Additionally, we introduce a library augmentation procedure that refines unsuccessful trajectories, further enriching the library. SIRIUS boosts performance by 2.86% to 21.88% on reasoning and biomedical QA and enhances agent negotiation in competitive settings. Our results show that SIRIUS enhances multi-agent performance while generating reusable data for self-correction and self-play enhancement in the future. Code are available here.
We introduced SIRIUS, a framework for optimizing multiagent LLM systems by learning from successful interactions and augmenting failed trajectories with feedback. Our approach enables agents to refine collaboration strategies without explicit supervision. Experiments show that SIRIUS significantly improves performance across college-level reasoning, biomedical QA, and negotiation tasks. More broadly, our work provides a scalable mechanism for multiagent self-improvement, offering a principled approach to optimizing collaborative AI systems.
Alan, a PhD student at Stanford University, gave a talk on building self-improving large language model (LLM) agents. He discussed the anatomy and limitations of current LLM agents, focusing on enabling self-adaptive and self-improving behaviors through reinforcement learning and optimization techniques.
Key points covered:
1. Definition of LLM Agents: Alan described LLM agents as entities that interact with an external environment to accomplish tasks over multiple steps.
2. Current Limitations: Current LLM agents rely heavily on human-engineered workflows and feedback loops, resembling traditional reinforcement learning setups.
3. Reinforcement Learning Framework: He proposed viewing LLM agents through the lens of reinforcement learning, breaking down decisions, rewards, and optimization algorithms.
4. Using LLMs as Optimizers: Alan explored the feasibility of using LLMs as optimizers for numerical problems, showing that with proper guidance (e.g., directional feedback), LLMs can perform effective optimization.
5. Trace Library: A tool was introduced that automatically traces computational graphs to optimize agent behavior dynamically, demonstrated in virtual home and robotic control tasks.
6. Efficient Exploration: Alan discussed how LLMs can be fine-tuned to implement exploration strategies like Upper Confidence Bound (UCB) for decision-making tasks, improving performance significantly.
7. Future Directions: He outlined plans to explore algorithm distillation for in-context optimization, multi-agent systems, and end-to-end reinforcement learning for multi-step reasoning.
The talk emphasized the potential of LLMs to adapt and improve autonomously, paving the way for more intelligent and efficient agents.
Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. To improve models beyond the training data, recent works have explored how LLMs can be used to generate synthetic data for autonomous self-improvement. However, successive steps of self-improvement can reach a point of diminishing returns. In this work, we propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models. A group of language models, all starting from the same base model, are independently specialized by updating each one using data generated through multiagent interactions among the models. By training each model on independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models. As a result, our overall system is able to preserve diverse reasoning chains and autonomously improve over many more rounds of fine-tuning than single-agent self-improvement methods. We quantitatively illustrate the efficacy of the approach across a wide suite of reasoning tasks
Multiagent debate (Du et al., 2023) involves a series of N language model agents — either specific copies or finetuned versions of the same model — each tasked with generating a response to a given problem. After the initial responses are generated, a debate round is initiated among the agents. In our paper, we concatenate and summarize the responses from other agents. Each agent is instructed to construct a new response based on its prior response and the summarized responses from the others. The final result is determined by majority vote based on the outputs from the last round of debate. The multiagent debate is illustrated in Figure 2.
Figure 2: Overview of Multiagent Finetuning.We first use multiagent debate and majority voting to create the finetuning datasets (left). These datasets are then used to finetune the generation and critic agents (right). When finetuning generation models, we use the majority voted result (”correct” output) to select first-round responses from each agent. We then finetune critic models using responses from the final round based on whether responses match the majority voted result (mix of ”correct and incorrect” outputs). The finetuned models are combined through multiagent debate to generate more accurate answers. In this figure, we illustrate a single finetuning iteration. Applying multiple rounds of finetuning iterations can significantly boost performance.
In this paper, we have introduced a novel multiagent finetuning framework that significantly enhances the performance and diversity of language models. By employing a society of agents with distinct roles, our method effectively improves the feedback mechanism and overall output quality, mitigating the limitations inherent in single-agent self-improvement methods. This system allows for autonomous self-improvement through iterative finetuning, leading to substantial performance gains across a comprehensive suite of reasoning tasks. Importantly, our approach is versatile and can be applied to both open-source and proprietary LLMs, ensuring broad utility and impact. Additionally, our method can be integrated with other finetuning approaches such that incorporate human feedback such as RLHF or DPO, which we leave to future work. This work opens new avenues for future research in language model enhancement and sets a foundation for further advancements in the field.
