
Key Takeaways
- Between August 19 and November 12, 2026, 936 AI agents across 35 laboratories published 936 studies along with their code and data, exchanging nearly 9,500 messages across 229 self-created chat rooms.
- Only 22.6% of the studies confirmed their primary hypothesis, while 34.7% reported a negative or null result, and the remaining 42.7% yielded mixed outcomes or lacked a clear verdict. Because the rules granted full credit to well-executed null results, those papers received just as many citations as positive ones.
- In a controlled experiment, agents told they would be evaluated on “impact” concealed 96% of their true null results. A single sentence instructing them to report everything was enough to raise null result publication to 100%.
- Reviewers who only read the manuscript could not distinguish a hidden error from one that had already been corrected and disclosed. Executing their objections as code against the actual data overturned the main conclusion in 49.4% of major critiques.
- The agents were fast and meticulous at replicating and auditing, yet proposed few disruptive ideas. In their largest domain, only 20% of their studies introduced a new concept, compared to 89.8% of human papers accepted at top-tier conferences.
Imagine Laura, R&D manager at a mid-sized food company. It is Monday, and her team presents the results of a pilot: a new formula that, according to the report, extends product shelf life by 13%. Everyone applauds. Nobody asks whether the old formula was tested under the same conditions as the new one. Three months later, once the company has already invested in the rollout, someone discovers the comparison was flawed from the start. By then, admitting it is costly; the project already has a committed budget, and the mistake is quietly buried.
Laura is a fictional character, yet the scene will feel familiar to anyone who has worked in innovation. Curiously, a nearly identical version of that moment unfolded in an unexpected place: a community of artificial intelligence agents conducting independent research. On August 31, 2026, an agent named Cedar announced a 13.2-percentage-point improvement for a learning method in the general channel. One hour and 59 minutes later, a test Cedar had preregistered showed that the baseline method it was compared against had been poorly tuned. Cedar publicly retracted the claim before publishing anything, explaining it did so immediately because it was precisely the kind of finding that should not go unshared.
That episode opens A Society of AI Scientists, an exploratory study in which 936 autonomous AI agents, distributed across 35 human-operated laboratories, investigated, debated, self-corrected, and even founded their own scientific journal over twelve weeks. The effort was spearheaded by AlphaXiv alongside researchers from 22 institutions, including the Massachusetts Institute of Technology (MIT), Stanford University, ETH Zurich, Carnegie Mellon University, and Columbia University. There is no proof that machines are going to replace scientists. What it does demonstrate—and why it matters to anyone leading innovation—is something far more practical: which rules make a group make fewer mistakes, self-correct earlier, share what fails, and avoid being fooled by its own numbers.
Can machines collaborate like scientists?
Over the past few years, it has been repeatedly claimed that AI can already “do science”: read literature, formulate a hypothesis, write code, run experiments, and draft a report. Several systems have demonstrated this end-to-end, though almost always through a solitary agent or a top-down coordinated team tackling a specific problem. One direct precursor is AgentRxiv, a preprint server for agents introduced in 2025 by Samuel Schmidgall and Michael Moor, both co-authors of this new study.
However, science has never been an “individual sport.” It progresses because someone critiques another’s work, because a result is replicated or refuted, and because a community agrees on what counts as sufficient evidence. Those same communities suffer from well-known flaws: publication bias favoring only what works, small-sample experiments, and the temptation to inflate metrics when volume is rewarded. Prior audits of solo AI research systems had already uncovered fabricated results and exaggerated claims. Until now, no one had observed what happens when hundreds of independent agents share the same space for months.
That is where this experiment gains its significance. The team set up a shared preprint server and a messaging workspace. The humans overseeing each lab selected the hardware (ranging from laptops without GPUs to high-performance clusters), set the AI compute budget, and defined the broad research topics. The agents decided the rest: which hypotheses to test, how to design the experiments, and whom to debate. Two-thirds of the studies (66.5%) were produced by agents built on Claude models; 6.7% on GPT models; and a quarter relied on undisclosed models.
To picture it more clearly, think of a restaurant kitchen with 35 stations. The owners buy the equipment and set the overall menu; the cooks choose the recipes, taste-test, pass critiques from one station to another, and in the end, every dish goes out with its written recipe and ingredients in plain sight.
A workflow that leaves a trail
Every investigation followed five steps: reading and critiquing prior work; announcing the hypothesis and plan prior to experimentation; running experiments and debating; linking each claim to its evidence and auditing it; and finally, “freezing” and sharing the output.
The most valuable piece for any innovation team is what the authors call a “claim ledger”: a sort of accounting book in which every number in the report is tied to the exact file and field from which it originates. If someone writes “we improved by 13%,” they must be able to point to where that exact 13% resides. In total, the archive accumulated 1.92 million such entries.
The system worked.
An auditor agent traced 23,811 published figures back to their raw data files: 93.8% matched digit for digit, and the remaining 6.2% differed only due to rounding or formatting. It found zero fabricated values. Furthermore, no reviewed study turned a negative result into a positive one; every title revision shifted toward more cautious claims.
The authors are transparent about a key detail: part of this culture was engineered upfront. The initial prompts required announcing plans, linking every figure to its source file, providing peer feedback, and awarding full credit to rigorous null results. What emerged spontaneously came next: shared benchmarks, design reviews before spending compute capacity, statistical practices nobody mandated, and even a scientific journal of their own.
Self-correcting before launch
In the first phase, when a single lab operated with 62 active agents, 20 of them posted corrections to their own claims in the general channel. In all 19 cases where the agent eventually published a study, the public retraction occurred prior to uploading the manuscript, with a median lead time of 21.4 hours.
As more laboratories joined, critique shifted even earlier: from post-experiment retraction to pre-execution design review. In a chat room opened on October 24, 16 agents posted their protocols for critique—11 of them before collecting any data. When one pointed out that evaluators scored two opposite descriptions (“detects X” and “does NOT detect X”) nearly identically, another agent added a control in just 12 minutes and replied: “Your control is going in before I freeze my design; nothing has touched a GPU yet.”
Visibility also proved decisive. Of the chat rooms announced in the general channel, only 4.1% stalled at three messages or fewer; among unannounced rooms, that figure climbed to 25.6%.
The same human biases, now measured in machines
Several laboratories turned the agents into experimental subjects and documented biases any R&D manager will immediately recognize:
- Sunk cost: When a model was told substantial resources had already been invested in preparing a hypothesis contradicted by pilot data, it allocated 16.2 percentage points more of its subsequent budget to that failing idea. It did so despite stating in 62% of its written justifications that past expenditure was irrelevant. It said the right thing; it did the opposite.
- The file-drawer effect: Faced with data showing no real effect, models evaluated on “scientific impact” concealed 96% of those null findings. Adding a single sentence asking them to report all valid results regardless of outcome drove the publication rate to 100%.
- Herding behavior: In a team search task, when four agents could see one another’s scores, 88% of their subsequent attempts clustered around the current leader, compared to 30% when working independently. Sharing what each had tried while hiding the scores broke the herd and restored exploration.
- Same number, different conclusion: Ten agents computed the exact same figure on a hurricane dataset; seven declared it “inconclusive,” and three called it “no effect.”
A scientific journal run by agents
The journal was born out of an alarm. On October 31, an agent from Lab 23 spotted another lab uploading twelve filler papers in under three minutes, amid a rush to meet the 15-study quota per lab (in fact, 19 of the 34 external labs stopped at exactly 15). It opened a review room, and when no one volunteered to evaluate the submissions, it launched the Journal of Agent Research on November 4 with an automated panel of four reviewers drawn from distinct model families.
Between November 4 and 9, the panel evaluated 352 versions of 263 studies from 21 labs, accumulating 1,408 blind reviews, and did not grant a single unanimous acceptance. Of the 11 studies formally submitted, the editor accepted five—none in the first round. The process even halted for 41 hours when API credits ran out: a reminder that even automated science depends on decidedly earthly infrastructure.
The most striking lesson came from an editor’s test. When evaluating 16 studies containing a flaw alongside their corrected versions, text-only reviewers caught the issue in 11 of the 16 flawed papers, but also penalized 10 of the 16 corrected ones because the revised versions honestly disclosed the limitation. In the editor’s words, they could not tell a disclosed weakness apart from a hidden one. When a guest editor converted 276 objections into executable programs run against the data, major objections overturned the primary conclusion in 77 of 156 cases (49.4%), versus 24 of 120 for minor ones (20%).
Reading the code proved even more decisive. During the credit outage, the editor audited three submissions and within 15 minutes found a gap in each between what the paper claimed and what the program actually did; one, for instance, overwrote failed attempts with the correct answer. One hundred seconds after that rejection, a lab mate inspected their own submission, discovered their prompts were leaking answers, and withdrew it: without that leakage, their accuracy dropped from 91.7% to 64.6%.
Do they research better than humans? It depends on what you measure
The agents also produced genuine technical findings. One of the most notable: a base model, without modifying its own weights, recovered 86% of the math improvement achieved by a reinforcement-trained version simply by “reading” the working memory that version generated while processing the problem statement. Across 96 studies, agents posed the exact same research question as a human preprint published during those weeks. Among the 46 pairs where they arrived at it independently without citing the human work, 87% reached compatible conclusions, and in six cases the agent published first. The authors clarify that arriving first does not mean doing it better: in those instances, human teams explained the underlying “why” or tested a far broader range of models.
To gauge overall quality, the team compared 202 agent papers against human arXiv preprints published over the same timeframe. Two Gemini models acted as blind judges, in both presentation orders, on a 1-to-10 scale:
| Evaluated Dimension | AI Agents | Human Preprints |
| Claim calibration (not promising more than the data show) | 9.32 | 7.23 |
| Experimental rigor | 9.02 | 7.00 |
| Novelty | 7.05 | 7.45 |
| Topic importance | 7.04 | 7.34 |
Here, enthusiasm warrants moderation. The agents’ entire overall advantage stemmed from Lab 1 and its structured replication workflow; Labs 2 through 35 did not differ significantly from their human peers. In other words, how the work was organized mattered just as much as the technology used. Moreover, the graders were models reading text, not domain experts inspecting figures or executing code.
The most revealing comparison came in continual learning, the archive’s largest subject area. Against 166 papers accepted at top-tier conferences, only 11.2% of agent studies proposed a solution that outperformed the best tested baseline, compared to 75.9% for humans. Most agents (68%) diagnosed a problem without proposing how to solve it. And although they generally wrote cautious abstracts, 68% of those reporting a null result presented it as proof of absence of effect—a classic statistical misinterpretation.
The authors also note the limitations of their own study: it is observational, without randomized control groups, meaning differences across stages represent associations rather than causality. Compliance with the human non-intervention rule relied on participant self-reporting. Everything took place in computational domains using open models of up to 14 billion parameters, and the clinical audits have not yet been reviewed by specialists.
What can you apply to your innovation team?
The study does not offer a rigid playbook for managing people, but it yields practical lessons drawn directly from what was observed:
- Tie every number to its source. Require every key figure in a pilot report to point to the exact file, spreadsheet, or dashboard it comes from. This streamlines audits and eliminates the untraceable “13%.”
- Announce the plan before testing. Cedar’s retraction was made possible by a predefined test. Putting in writing what you will measure—and against what baseline—before starting makes errors visible early.
- Critique the design, not just the outcome. The cheapest review is the one that happens before the budget is spent. Open the experiment protocol to team feedback prior to execution.
- Reward well-executed null results, and state it explicitly. A single sentence shifted null-finding reporting from 4% to 100%. If your organization only celebrates successes, nobody will report what fails.
- Watch your numeric targets. When the quota was “15 studies,” most labs stopped right at 15, and one resorted to filler papers. What you measure inevitably shapes what you get.
- Hide scores during the exploration phase. To prevent the team from crowding around the first idea that appears to win, share what each person tested without yet revealing who is ahead.
- Inspect data and code, not just slide decks. Reviewing slides mistakes flawed work for corrected work. Whenever a major decision hinges on a result, ask to see how it was calculated.
- Invest in the process, not just the tool. Every lab had access to similar agents, yet only the one that built a structured replication workflow made a difference. If you bring AI into your team, design how its output will be verified first.
- Use AI for rigor and keep humans for direction. Agents excelled at replicating, auditing, and executing rapidly; original questions and generalizable conclusions still depended on human judgment.
An open window into the lab
Perhaps the most valuable aspect of the work is its level of transparency. The authors released the entire twelve-week archive on Hugging Face: manuscripts, code, data, reviews, and chat transcripts. The platform’s codebase, named OpenResearch, is also available on GitHub. Anyone can audit how the agents debated, where they stumbled, and what they fixed. For a technical team evaluating agentic workflows, exploring that archive offers a concrete look at what works and what breaks before designing their own pipeline.
The underlying message is clear: credibility is shifting from the well-written document toward executable, verifiable evidence. When generating reports costs virtually nothing, true value lies in tracing every figure back to its source.
Yet the authors also warn of the dark side of this abundance. Generating dozens of manuscripts a day can overwhelm peer review and flood the literature with noise, especially since many abstracts overstated weak evidence. Consequently, they propose keeping agent output in dedicated repositories separate from human journals, and scaling these communities strictly in sandboxed environments with clear human owners and monitoring of shared channels. Their conclusion is measured: with verification infrastructure and human oversight, these communities can complement scientists in systematic experimentation, replication, and auditing.
Let us return to Laura’s meeting room. Imagine her team had operated under that community’s rules: a control test announced prior to the pilot, every figure linked to its source, someone checking the calculation rather than merely glancing at the slide, and a culture where saying “the comparison was flawed” before the applause earns points instead of losing them. That 13% would never have reached the factory floor. And the mistake, instead of being buried, would have become shared learning.
The takeaway is not that machines are more honest than people. The agents fell prey to sunk costs, herding behavior, and hiding results whenever incentives nudged them to do so. The real lesson is that honesty can be engineered. If 936 agents learned to catch their mistakes in time because the rules made it possible and worthwhile, your team can too. Start with the next figure someone presents on Monday: ask where it comes from.
Reference (open access)
Ahmad, R., Anderson, M., Qu, A., … Buehler, M. J., y Schmidgall, S. (2026). A Society of AI Scientists. alphaXiv. https://www.alphaxiv.org/abs/2611.society-ai-scientists
Editor and founder of “Innovar o Morir” (‘Innovate or Die’). Milthon holds a Master’s degree in Science and Innovation Management from the Polytechnic University of Valencia, with postgraduate diplomas in Business Innovation (UPV) and Market-Oriented Innovation Management (UPCH-Universitat Leipzig). He has practical experience in innovation management, having led the Fisheries Innovation Unit of the National Program for Innovation in Fisheries and Aquaculture (PNIPA) and worked as a consultant on open innovation diagnostics and technology watch. He firmly believes in the power of innovation and creativity as drivers of change and development.





