How to evaluate the performance of AI models in mathematics?

On 15 March 2016, the fifth Go match between Lee Sedol, the 2nd most successful player in history, and AlphaGo, a computer programme developed by Deepmind, a British company acquired by Google two years earlier, ended. AlphaGo won the competition, with 4 wins to 1 defeat.

Speaking of move 37 in the second game, Sedol declared: “I thought AlphaGo was based on probability calculations, and that it was just a simple machine. But, when I saw that move, I changed my mind. Assuredly, AlphaGo is creative. That move was truly creative and beautiful.”

The game of Go has long been considered a significant challenge for AI: the game presents 10 to the power of 170 possible board configurations, which is one googol (10 to the power of 100) times more than chess. It’s even more than the number of atoms in the known universe.

AlphaGo is an AI system combining two neural networks (Policy Network and Value Network) and advanced search algorithms. AlphaGo's training leveraged Reinforcement Learning: AlphaGo played millions of games against itself.

Despite its astronomical mathematical complexity, the game of Go “simply” consists of a 19x19 board and 361 stones. This finite universe intuitively seems simpler for a machine to master than moving within and acting upon the physical universe (robotics), or... mathematics, which Banach said remains “the most beautiful and powerful creation of the human mind”. Until now? That's what we will try to discover.

In 2021, Dan Hendrycks et al. published a paper titled “Measuring Mathematical Problem Solving With the MATH Dataset”. MATH is a dataset of 12,500 mathematical problems, along with their solutions and associated proofs.

The MATH problems cover 7 subjects, including geometry, linear algebra, and number theory, and are classified into 5 difficulty levels. Human performance on MATH is not perfect: a computer science PhD student would achieve a performance of about 40%, whereas a triple gold medallist at the International Mathematical Olympiads would achieve a performance of about 90%.

And AI models? Their performance ranges from 3.0% to 6.9%, with a maximum of 15% on the simplest problems. The models tested were GPT-2 (from 100 million parameters to 1.5 billion) and GPT-3 (from 13 to 175 billion parameters), models now outdated.

In 2021, the authors concluded that, unlike many other text-based tasks, "scaling" transformer-based models (which are the foundation of Generative AI) do not solve the MATH benchmark.

And today?

Less than 4 years later, the situation is somewhat different: the OpenAI's o1 model, published in September 2024, would achieve a performance of 94.8% on the MATH benchmark. This benchmark is now considered saturated. o1 also achieves a performance of 83.3% on the problems of the AIME 2024 competition, a benchmark of superior difficulty.

AIME (for American Invitational Mathematics Examination) is a dataset comprising problems from the competition of the same name, which has been running since 1983. It is a 3-hour exam with 15 questions, intended for the best American high school mathematics students. The associated problems are complex, require creative approaches, and all have an integer solution between 0 and 999.

OpenAI's “o” family models are the first in their category, with scaling on test-time compute. Scaling at test-time means, if one reasons anthropomorphically, that the model takes time to reflect before giving an answer. More concretely, it generates hidden tokens, where it develops long reasonings, before generating the definitive answer that will be provided to the user. This innovation therefore allows models to be considered as Type II thinking systems, and not Type I.

More recently, the Chinese company DeepSeek published a eponymous reasoning model, also benefiting from this innovation. According to this article, DeepSeek-R1 would achieve “only” a 65% performance on the 2025 edition of AIME.

Even more recently, on 31 January 2025, OpenAI released two new models: o3 and o3-mini. o3-mini would achieve an 80% performance on AIME 2025, and o3 would achieve a 96.7% performance, thus saturating this benchmark!

“Scaling at test-time” is a promising new research dimension, and should allow, from 2025 and beyond, the development of AI models with increasingly robust and complex reasoning. However, to attest to this, more demanding new benchmarks would need to be created. This has been the case for a few months, with FrontierMath.

In November 2024, the Epoch AI research institute published FrontierMath, a benchmark containing hundreds of expert problems, which the best AI models (of the time) solve in only 2% of cases.

These problems were written by around sixty mathematicians, including Olympiad question setters and Fields medallists. Deliberately innovative and unpublished, these problems aim to test the models' real mathematical understanding. They are also “robust”, with a probability of less than 1% of randomly guessing the correct answer.

According to Epoch AI, the models were tested under the most favourable conditions possible: extended reflection time, and the ability to experiment and iterate via a Python environment where they can execute code to test hypotheses and check their intermediate results. In November 2024, only Gemini 1.5 Pro and Claude 3.5 Sonnet solved more than 1% of the problems. In mathematics, a victory for the (best) humans, then?

To date, yes. However, OpenAI's o3 model, published shortly after this benchmark, would achieve a performance of 25%, indicating rapid progress in a few months.

According to this source, FrontierMath would be broken down into three levels of problems, with Tier 1 corresponding in difficulty to the International Mathematical Olympiads. o3 would therefore be able to solve most of the problems at that level, but not beyond.

But let's go back to DeepMind, now a Google subsidiary and research lab. Less publicised than OpenAI, they are not to be outdone in this race for innovation. In July 2024, they published the models AlphaProof et AlphaGeometry 2, dedicated to formal mathematical reasoning and based on Reinforcement Learning. These models solved 4 out of 6 problems from the 2024 International Mathematical Olympiad, a performance that earned a silver medal in this edition. The equivalent of a 59th place out of 609 participants.

Finally, DeepMind states that for this edition, they tested a natural language reasoning system, based on Gemini, which avoids the need to translate the problems presented into the formal language specific to AlphaProof and AlphaGeometry 2. The results are said to be “very promising”. The 2025 edition of the Olympiads should hold its share of surprises.

Given the speed of publication of new AI models, and the recent and massive investments in the field, it is clear that AI has not said its last word.

Regarding benchmarks, innovations are also to be expected: Epoch AI recently announced that they are working on a suite of Tier 4 problems, problems that would require the collaboration of entire mathematics departments to be solved.

The most optimistic about the future of AI argue that increasing the volume of training data and the computational power provided will allow AI model performance to increase logarithmically (and indefinitely), according to the scaling laws. This scaling can occur across several different dimensions, as previously seen: pre-training, post-training, and, more recently, on “computations at inference time”.

In December 2024, OpenAI invited Terence Tao, Fields Medallist 2006, to a conference on the future of mathematics with “reasoning” models. He is optimistic about the use of AI in general for mathematics research, where models will play the role of research assistants, capable of performing some of the tasks that are currently entirely manual.

However, he argues that mathematics will never be entirely solved by AI: "Even if AI becomes capable of doing all the mathematics we do today, that simply means we will move on to a higher type of mathematics." One thing is certain, AI has an important role to play, and will accelerate the pace of discoveries in this field.