ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) is an AI benchmark released by François Chollet — known as the author of Keras — alongside his 2019 paper “On the Measure of Intelligence.” It is now run by the non-profit ARC Prize Foundation, and it uses tasks such as colored grid puzzles that are easy for people yet hard for AI to measure how efficiently a system adapts to problems it has never seen. Its focus is not the amount of knowledge or the scale of pre-training, but the ability to work out new rules on the spot (fluid intelligence), which is why it is widely cited as one of the leading indicators of progress toward AGI.
Key Features
- Measures intelligence as skill-acquisition efficiency: Rather than asking how much a model has memorized, it asks how quickly a system can learn an unfamiliar task from a handful of examples. In Chollet’s definition, intelligence is precisely that learning efficiency
- Tasks that do not depend on cultural knowledge: Every task is designed to be solvable using only “core knowledge priors” — the cognitive basics such as color, number, symmetry and objectness that people acquire early in life. Because differences in language or domain expertise do not affect the outcome, humans and AI can be compared on equal footing
- ARC-AGI-1 (2019): The original version, consisting of 1,000 tasks and created before the rise of LLMs. It is known for the fact that scores barely moved for years, until test-time adaptation methods appeared in late 2024
- ARC-AGI-2 (2025): Comprises 1,000 training tasks plus a calibrated public evaluation set of 120 tasks, a semi-private set of 120 tasks for the Kaggle leaderboard, and a private set of 120 tasks for final standings. It was calibrated through in-person testing with more than 400 participants in San Diego, and every task has been solved by at least two humans within two attempts
- Explicit measurement of cost efficiency: From ARC-AGI-2 onward, results report cost per task alongside accuracy. The design principle that intelligence means finding the solution efficiently rather than exhaustively is built into the metric itself
- ARC-AGI-3 (2026): Released on March 25, 2026, this is the first interactive reasoning benchmark. Instead of static input-output pairs, it places AI agents in game environments with no instructions, no stated goals and no explicit rules, requiring them to explore, build a world model, set goals, and plan and execute on their own. At launch, every frontier model scored below 1%, while humans solved all of the environments
- Public leaderboards and competitions: Verified and community leaderboards are published, and the annual ARC Prize competition runs on Kaggle with prize money
Pricing
| Item | Cost | Details |
|---|---|---|
| Datasets (training and public evaluation) | Free | Available to anyone from the official site and GitHub |
| Trying tasks and viewing leaderboards | Free | You can solve the actual tasks yourself in the browser |
| Entering ARC Prize 2026 | Free (submitted via Kaggle) | The ARC-AGI-3 track carries $850K in total prizes ($700K grand prize, $75K in top-score awards, $75K in milestone prizes). Including the ARC-AGI-2 track, the overall pool exceeds $2 million |
| Running models for evaluation | Paid by the participant | If you use commercial APIs to compete, the inference cost is your own |
Pricing is current as of August 2026. Please check the official site for the latest information.
Pros & Cons
✅ Pros
- Because it measures adaptability to unseen tasks rather than pre-training scale or knowledge volume, it makes a model’s raw reasoning ability easier to judge
- The tasks are visual grid puzzles, so anyone can solve a few and feel the difficulty firsthand without specialist knowledge
- Datasets, leaderboards and documentation are all public, so individuals can use them for research and verification
- Reporting cost alongside accuracy closes the loophole of simply throwing money at the problem
- With ARC-AGI-3, the evaluation axis has expanded from static question answering to the behavior of agents
⚠️ Cons
- It is a research benchmark, not an AI tool for daily work. It is not meant to make everyday tasks more efficient
- A score does not translate directly into practical performance. A high ARC-AGI result is no guarantee of strong writing or coding ability
- The tasks are highly abstract, so interpreting the results requires reading the papers and official documentation
- Competing seriously means meeting Kaggle’s compute constraints and open-source requirements, which takes effort
Comparison with Similar Services
| Comparison | ARC-AGI | Humanity’s Last Exam | SWE-bench | MMLU |
|---|---|---|---|---|
| What it measures | Adaptation to unseen tasks and learning efficiency | Hardest expert-level knowledge and reasoning | Bug fixing in real repositories | Broad general knowledge |
| Task format | Colored grid puzzles / interactive game environments | Hard questions written by domain experts | Real GitHub issues with matching tests | Multiple-choice exam questions |
| Human baseline | Ordinary people solve nearly all of them | Extremely hard even for experts | Solvable by experienced engineers | Varies widely by field |
| Effect of prior knowledge | Designed to have almost none | Large | Large | Large |
| Main use | Measuring progress toward AGI | Probing the ceiling of frontier models | Evaluating coding agents | Comparing general knowledge |
Who Is It For
- People who want to understand how capable AI really is today, including the context behind benchmark numbers
- People who want to follow the AGI debate through measurable indicators rather than marketing claims
- Researchers and developers working on reasoning and generalization who want to validate methods on a public dataset
- Engineers who want to take on test-time adaptation or agent design through a competition with prize money
- Anyone who needs concrete examples of where humans and AI differ, for teaching or talks
Summary
ARC-AGI stands apart from other major benchmarks in that it tries to measure AI by efficiency of adaptation to the unknown rather than by volume of knowledge. The fact that frontier models struggle with tasks people solve in seconds is a concise statement of what current AI still lacks. By stepping into interactive agent evaluation with ARC-AGI-3, that question has widened further into whether a system can act and learn. It is not a tool for practical work, but if you want to see the state of AI progress with your own eyes, the fastest route is to solve a few of the actual tasks on the official site.