Opens in a new window Opens an external website Opens an external website in a new window
This website utilizes technologies such as cookies to enable essential site functionality, as well as for analytics, personalization, and targeted advertising. Privacy Policy
Accept Deny Non-Essential Manage Preferences
Storage Preferences

Quick Search anywhere
Enter Search Term
Quick Search in Journals
Enter Search Term
Quick Search in Journals
Enter Search Term
Quick Search in Journals
Enter Search Term
Quick Search in Journals
Enter Search Term
Quick Search in Journals
Enter Search Term
Quick Search in Journals
Enter Search Term
Searching:
Anywhere
AnywhereScienceScience AdvancesScience ImmunologyScience RoboticsScience SignalingScience Translational Medicine
Quick Search anywhere
Enter Search Term
Main content starts here
Can researchers bring order to the confusing world of AI ‘benchmarks’?
10 Nov 2026
10:55 AM ET

JASON SOLO/THE JACKY WINTER GROUP
Share:
Play Pause
Skip backwards Go ten seconds backward Skip forwards Go ten seconds forward
Progress
1.0x
Mute Unmute
Volume is at 50%
00:00
05:12
1.0x
Say you are trying to decide which artificial intelligence (AI) tool would be best for translating a research paper from Spanish to English. Browsing the web, you discover one analysis gives the best grade to Google’s Gemini large language model, but another ranks Translated’s Lara 3 at the top—whereas a third gives honors to an open model from Nous Research.
Welcome to the chaotic and confusing world of AI benchmarks—tests designed to measure everything from a model’s speed and accuracy to its ability to bypass safeguards. Now, however, researchers are working to bring greater transparency and reproducibility to benchmarking, launching several initiatives aimed at making it easier to understand what the scores actually mean and whether they can be trusted.
The need is pressing, says Avijit Ghosh, a policy researcher at Hugging Face, a platform for sharing AI tools. Benchmarkers often don’t fully reveal their methods, making it difficult for others to replicate or interpret the results. For the paper translation task, for example, accuracy scores for the three top models ranged from 40% to 94%, with little guidance for interpreting them. And testers using the same benchmarking tool can get different results, depending on their methods. “Benchmarks are very fragile,” says Savannah Thais, an AI researcher at Hunter College. “If you change the punctuation in benchmark questions, you can get different results.”
The latest news, commentary, and research, free to your inbox daily
For scientists, the disorder is more than just a headache. Benchmarks can influence which AI tools they use in research projects. There are financial implications, too. It can cost thousands of dollars or more to figure out whether a benchmark result is trustworthy—and many labs simply can’t afford to replicate scores. “Running or rerunning evaluations is extremely infeasible,” Ghosh says.
One effort to address the problem is Every Eval Ever (EEE), a public database of crowdsourced evaluation results launched earlier this year by Ghosh and more than 600 others. The repository, which is part of the EvalEval Coalition, a volunteer community working to improve AI evaluation, makes it easier to share standardized information about benchmark results and logs. It documents exactly when a benchmark was run, for example, and what computing hardware was used. At workshops and conferences introducing EEE, “we had more interested researchers than chairs in the rooms and were beyond capacity,” says Jan Batzner, an AI researcher at the Weizenbaum Institute and the Technical University of Munich.
The EvalEval Coalition is also building tools to help model developers share data, says Anka Reuel, an AI governance researcher at Stanford University who is part of the coalition. For example, it has built programs that automatically convert benchmark results into EEE’s format, potentially saving hours of labor.
Ziang Xiao, a human-computer interaction researcher at Johns Hopkins University, is tackling another part of the problem: transparency about what the models actually produce. He’s collecting the specific responses models give to benchmark tasks or questions. Such detailed data can be critical to understanding exactly how good a benchmark is at evaluating what it claims to measure, Thais says. So far, Xiao’s OpenEval collection contains about 10 million responses from popular benchmarks, which are also shared to the EEE database.
For researchers seeking a crowdsourced verdict on benchmarks, the EvalEval Coalition has released Evaluation Cards that visually summarize evaluations done by others. The goal is to show users, at a glance, “how complete and reproducible [an evaluation] is,” Thais says. The cards also identify what kinds of tasks the benchmark is intended to test and its potential limitations and biases. As of October, the Evaluation Cards repository included more than 320,000 results, covering more than 1500 benchmarks and 9700 models.
It’s unclear whether these efforts will bring lasting order to benchmarking. Similar past efforts, including so-called EvalCards and Audit Cards, didn’t see widespread adoption.
Yet the importance of benchmarking is likely to remain. Companies use benchmark results to market their products, and the European Union is now collecting testing data in order to evaluate the risks posed by various models. Other governments have established AI security and safety institutes that use benchmarks to formulate policy recommendations, Ghosh notes. So far, however, governments have shown relatively little interest in requiring standardized public reporting of benchmark data.
“We don’t always value transparency super highly in AI research, not because people are necessarily bad actors, but because there are a lot of other incentives in this space to go quickly,” Thais says. But she hopes the new tools and databases “lower the barrier to entry” by making it easier to share data. “I’m hopeful,” Thais says, “that this helps get more people on board.”
Science’s AI in Science reporting initiative is supported by Ray Rothrock & family.
doi: 10.1126/science.zf5bdog
Scientific CommunityArtificial intelligenceScience’s AI in Science Reporting Initiative
Ananya is an independent journalist and translator based in Bengaluru, India.

10 Nov 2026
By
Rodrigo Pérez Ortega

10 Nov 2026
The ‘Pathogen Hunters’ track microbes around the world. But are their data real?
By
Jon Cohen

10 Nov 2026
By
Jeffrey Mervis
Get Science’s award-winning newsletter with the latest news, commentary, and research, free to your inbox daily.
10 Nov 2026 By
Jon Cohen
The ‘Pathogen Hunters’ track microbes around the world. But are their data real?
10 Nov 2026 By
Jeffrey Mervis
10 Nov 2026 By
Ananya
Can researchers bring order to the confusing world of AI ‘benchmarks’?

9 Nov 2026 By
Phie Jacobs
To deter bioterrorism, U.S. imposes new rules on ordering synthetic DNA

6 Nov 2026 By
Jocelyn Kaiser

29 Oct 2026 By
Sara Reardon

27 Oct 2026 By
Jocelyn Kaiser

20 Oct 2026 By
Kai Kupferschmidt
Books July 1998
Meeting Report December 2010
Working with the CTSA Consortium: What We Bring to the Table
Research Article November 2022
Generative and reproducible benchmarks for comprehensive evaluation of machine learning classifiers
Letter August 2017
Technology 18 Nov 2026
We didn’t evolve alone: How microbes shaped human adaptation
$
Please enter a valid amount
Single
Monthly
Email:
Please enter a valid email
Country:
[x] I also wish to receive emails from AAAS/Science and Science advertisers, including information on products, services and special offers which may include but are not limited to news, careers information & upcoming events.
Amid the unprecedented upheaval in U.S. science, hard-hitting, trustworthy reporting has never been more important. As part of the AAAS mission, Science has built a global award-winning network of reporters and editors that independently cover the most important developments in research and policy.
Your tax-deductible contribution plays a critical role in sustaining this effort.
If, at any time, you would like to stop or make changes to your monthly gift, please contact the Philanthropy Office at philanthropy@aaas.org or 202-326-6636.
Loading...
© 2026 American Association for the Advancement of Science. All rights reserved. AAAS is a partner of HINARI, AGORA, OARE, CHORUS, CLOCKSS, CrossRef and COUNTER.

Help News from Science publish trustworthy, high-impact stories about research and the people who shape it. Please make a tax‑deductible gift today.
✓
Thanks for sharing!
Site index)