The SignalToday's digest →

Can researchers bring order to the confusing world of AI ‘benchmarks’?

Science · original · full text captured 16 November 2026 · WhatsApp · Email

Opens in a new window Opens an external website Opens an external website in a new window

This website utilizes technologies such as cookies to enable essential site functionality, as well as for analytics, personalization, and targeted advertising. Privacy Policy

Accept Deny Non-Essential Manage Preferences

Storage Preferences

Quick Search anywhere

Enter Search Term

Quick Search in Journals

Enter Search Term

Quick Search in Journals

Enter Search Term

Quick Search in Journals

Enter Search Term

Quick Search in Journals

Enter Search Term

Quick Search in Journals

Enter Search Term

Quick Search in Journals

Enter Search Term

Searching:

Anywhere

AnywhereScienceScience AdvancesScience ImmunologyScience RoboticsScience SignalingScience Translational Medicine

Trending Terms:

Quick Search anywhere

Enter Search Term

Main content starts here

Can researchers bring order to the confusing world of AI ‘benchmarks’?

Can researchers bring order to the confusing world of AI ‘benchmarks’?

Several efforts aim to make torrent of AI testing results easier to understand and use

10 Nov 2026

10:55 AM ET

JASON SOLO/THE JACKY WINTER GROUP

Share:

Play Pause

Skip backwards Go ten seconds backward Skip forwards Go ten seconds forward

Progress

1.0x

Mute Unmute

Volume is at 50%

00:00

05:12

1.0x

Say you are trying to decide which artificial intelligence (AI) tool would be best for translating a research paper from Spanish to English. Browsing the web, you discover one analysis gives the best grade to Google’s Gemini large language model, but another ranks Translated’s Lara 3 at the top—whereas a third gives honors to an open model from Nous Research.

Welcome to the chaotic and confusing world of AI benchmarks—tests designed to measure everything from a model’s speed and accuracy to its ability to bypass safeguards. Now, however, researchers are working to bring greater transparency and reproducibility to benchmarking, launching several initiatives aimed at making it easier to understand what the scores actually mean and whether they can be trusted.

The need is pressing, says Avijit Ghosh, a policy researcher at Hugging Face, a platform for sharing AI tools. Benchmarkers often don’t fully reveal their methods, making it difficult for others to replicate or interpret the results. For the paper translation task, for example, accuracy scores for the three top models ranged from 40% to 94%, with little guidance for interpreting them. And testers using the same benchmarking tool can get different results, depending on their methods. “Benchmarks are very fragile,” says Savannah Thais, an AI researcher at Hunter College. “If you change the punctuation in benchmark questions, you can get different results.”

SIGN UP FOR THE AWARD-WINNING SCIENCE ADVISER NEWSLETTER

The latest news, commentary, and research, free to your inbox daily

For scientists, the disorder is more than just a headache. Benchmarks can influence which AI tools they use in research projects. There are financial implications, too. It can cost thousands of dollars or more to figure out whether a benchmark result is trustworthy—and many labs simply can’t afford to replicate scores. “Running or rerunning evaluations is extremely infeasible,” Ghosh says.

One effort to address the problem is Every Eval Ever (EEE), a public database of crowdsourced evaluation results launched earlier this year by Ghosh and more than 600 others. The repository, which is part of the EvalEval Coalition, a volunteer community working to improve AI evaluation, makes it easier to share standardized information about benchmark results and logs. It documents exactly when a benchmark was run, for example, and what computing hardware was used. At workshops and conferences introducing EEE, “we had more interested researchers than chairs in the rooms and were beyond capacity,” says Jan Batzner, an AI researcher at the Weizenbaum Institute and the Technical University of Munich.

The EvalEval Coalition is also building tools to help model developers share data, says Anka Reuel, an AI governance researcher at Stanford University who is part of the coalition. For example, it has built programs that automatically convert benchmark results into EEE’s format, potentially saving hours of labor.

Advertisement

Ziang Xiao, a human-computer interaction researcher at Johns Hopkins University, is tackling another part of the problem: transparency about what the models actually produce. He’s collecting the specific responses models give to benchmark tasks or questions. Such detailed data can be critical to understanding exactly how good a benchmark is at evaluating what it claims to measure, Thais says. So far, Xiao’s OpenEval collection contains about 10 million responses from popular benchmarks, which are also shared to the EEE database.

For researchers seeking a crowdsourced verdict on benchmarks, the EvalEval Coalition has released Evaluation Cards that visually summarize evaluations done by others. The goal is to show users, at a glance, “how complete and reproducible [an evaluation] is,” Thais says. The cards also identify what kinds of tasks the benchmark is intended to test and its potential limitations and biases. As of October, the Evaluation Cards repository included more than 320,000 results, covering more than 1500 benchmarks and 9700 models.

It’s unclear whether these efforts will bring lasting order to benchmarking. Similar past efforts, including so-called EvalCards and Audit Cards, didn’t see widespread adoption.

Yet the importance of benchmarking is likely to remain. Companies use benchmark results to market their products, and the European Union is now collecting testing data in order to evaluate the risks posed by various models. Other governments have established AI security and safety institutes that use benchmarks to formulate policy recommendations, Ghosh notes. So far, however, governments have shown relatively little interest in requiring standardized public reporting of benchmark data.

“We don’t always value transparency super highly in AI research, not because people are necessarily bad actors, but because there are a lot of other incentives in this space to go quickly,” Thais says. But she hopes the new tools and databases “lower the barrier to entry” by making it easier to share data. “I’m hopeful,” Thais says, “that this helps get more people on board.”

Science’s AI in Science reporting initiative is supported by Ray Rothrock & family.

doi: 10.1126/science.zf5bdog

Relevant tags:

Scientific CommunityArtificial intelligenceScience’s AI in Science Reporting Initiative

About the author

Author

Ananya is an independent journalist and translator based in Bengaluru, India.

More from news

10 Nov 2026

By

Rodrigo Pérez Ortega

10 Nov 2026

The ‘Pathogen Hunters’ track microbes around the world. But are their data real?

By

Jon Cohen

10 Nov 2026

By

Jeffrey Mervis

Sign up for ScienceAdviser

Get Science’s award-winning newsletter with the latest news, commentary, and research, free to your inbox daily.

Advertisement

ScienceInsider

10 Nov 2026 By

Jon Cohen

The ‘Pathogen Hunters’ track microbes around the world. But are their data real?

10 Nov 2026 By

Jeffrey Mervis

10 Nov 2026 By

Ananya

Can researchers bring order to the confusing world of AI ‘benchmarks’?

9 Nov 2026 By

Phie Jacobs

To deter bioterrorism, U.S. imposes new rules on ordering synthetic DNA

Advertisement

Briefs

6 Nov 2026 By

Jocelyn Kaiser

29 Oct 2026 By

Sara Reardon

27 Oct 2026 By

Jocelyn Kaiser

20 Oct 2026 By

Kai Kupferschmidt

Books July 1998

Meeting Report December 2010

Working with the CTSA Consortium: What We Bring to the Table

Research Article November 2022

Generative and reproducible benchmarks for comprehensive evaluation of machine learning classifiers

Letter August 2017

Technology 18 Nov 2026

We didn’t evolve alone: How microbes shaped human adaptation

Advertisement

$50

$75

$100

Other

$

Please enter a valid amount

Single

Monthly

Email:

Please enter a valid email

Country:

[x] I also wish to receive emails from AAAS/Science and Science advertisers, including information on products, services and special offers which may include but are not limited to news, careers information & upcoming events.

Support nonprofit science journalism

Amid the unprecedented upheaval in U.S. science, hard-hitting, trustworthy reporting has never been more important. As part of the AAAS mission, Science has built a global award-winning network of reporters and editors that independently cover the most important developments in research and policy.

Your tax-deductible contribution plays a critical role in sustaining this effort.

If, at any time, you would like to stop or make changes to your monthly gift, please contact the Philanthropy Office at philanthropy@aaas.org or 202-326-6636.

Loading...

© 2026 American Association for the Advancement of Science. All rights reserved. AAAS is a partner of HINARI, AGORA, OARE, CHORUS, CLOCKSS, CrossRef and COUNTER.

Support nonprofit science journalism

Help News from Science publish trustworthy, high-impact stories about research and the people who shape it. Please make a tax‑deductible gift today.

✓

Thanks for sharing!

Site index)