AgentCoder is a research-born open source framework that improves the accuracy of LLM code generation through a division of labor among multiple agents. Three agents work together: a programmer agent that writes code, a test designer agent that independently designs test cases, and a test executor agent that actually runs the tests and returns the results. Based on feedback from failing tests, the code is rewritten again and again. The paper “AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation” was published on arXiv in December 2023, and the research team led by Dong Huang at the University of Hong Kong has released the implementation on GitHub under the MIT license.
Key Features
- Division of labor across three specialized agents: The framework separates “the one who writes,” “the one who designs tests,” and “the one who runs tests.” The programmer agent generates code, the test designer agent creates test cases independently without looking at the implementation, and the test executor agent runs them and returns the results
- Test design kept independent from code generation: If the same model is asked to write both the code and the tests, it tends to write tests that suit its own code, which defeats the purpose of verification. AgentCoder avoids this bias by splitting test design into a separate agent
- Iterative optimization through failure feedback: When a test fails, the error is fed back to the programmer agent for correction, and the loop repeats. Think of it as automating the work of a human saying “this doesn’t run” and asking for a fix
- High accuracy on benchmarks: The paper reports 96.3% pass@1 on HumanEval and 91.8% on MBPP when using GPT-4. It also states that this consumed fewer tokens than the state-of-the-art methods of the time (90.2% on HumanEval / 78.9% on MBPP)
- Python implementation under the MIT license: You clone the repository, install dependencies with
pip install -r requirements.txt, set an API key in.env, and run the scripts in order (programmer → test designer → test executor). The license makes it easy to use for both research and commercial evaluation
Pricing
| Item | Cost | Details |
|---|---|---|
| AgentCoder itself | Free | Open source published on GitHub (MIT license) |
| LLM usage | Pay as you go | API charges from the LLM provider you use (OpenAI and others) apply separately |
Pricing is as of August 2026. AgentCoder itself has no paid plans. For pay-as-you-go LLM charges, check the official page of each provider. For the latest state of the implementation, see the official repository.
Pros & Cons
✅ Pros
- The software itself is free and the full source is readable, so you can verify its behavior and adapt it to your own use case
- The “generate → test → fix” loop is built in from the start, which visibly reduces obvious errors compared with one-shot generation
- The design of delegating test creation to an independent agent is a useful reference when building your own agent infrastructure
- The paper and the code are published as a pair, so you can follow the reasoning behind the architecture
⚠️ Cons
- This is a research prototype rather than a product: there is no GUI and no installer, and knowledge of Python and the command line is assumed
- It was built for benchmarks like HumanEval and MBPP that ask for a single function, so it does not directly apply to modifying an existing large repository
- Because the agents exchange messages repeatedly, the number of LLM API calls and the cost increase compared with a single generation
- Since publishing research results is the main purpose, continuous updates and support cannot be expected to the degree of a commercial tool
- The generated tests are not always correct, so a human still has to perform the final review
Comparison with Similar Services
| Item | AgentCoder | MetaGPT | ChatDev | AutoGen |
|---|---|---|---|---|
| Positioning | Research-born code generation framework | Multi-agent system modeling the software development process | Multi-agent system modeling a virtual software company | General-purpose multi-agent building framework |
| Main focus | Verification through tests and iterative fixes | Reproducing the flow from requirements to implementation | Reproducing a development process through role division | The design of agent-to-agent conversation itself |
| Target tasks | Function-level code generation | Small to medium application generation | Small application generation | General tasks, not limited to code generation |
| License | MIT | Open source | Open source | Open source |
| Runtime | Python scripts | Python | Python | Python / .NET |
Who Is It For
- People who want to understand how to improve LLM code generation accuracy, from both the paper and the implementation
- Engineers who want to add a “verification loop through automated testing” to their own code generation pipeline
- Developers who want to study a successful design before building their own multi-agent setup
- Researchers and students who want to run comparative experiments or benchmark evaluations of code generation methods
Summary
AgentCoder offers a clear answer to the limits of “let one LLM write the code and call it done”: make test design independent and run a verification loop. It is less a ready-to-use development tool than a foundation for learning the mechanism and adopting it into your own environment. If you are interested in multi-agent code generation, the paper and the repository are worth reading side by side.