Testing Rowan-Enabled Agents on DrugDiscoveryBench

by Eli Mann · Jul 21, 2026

Early this month, Scale Labs and Phylo released DrugDiscoveryBench, a benchmark designed to evaluate how well agents can perform early-stage drug discovery tasks. The benchmark consists of 82 expert-curated tasks across a variety of fields, including structural reasoning, patent mining, target ID & genetics, cheminformatics, molecular biology, and compound SAR/affinity ranking. The authors evaluated a variety of top models in a comprehensive report and open-sourced their benchmark code.

Careful readers of our newsletter and blog will recall that we've developed a skill to make it easy for scientific AI agents to use Rowan. We wanted to see how a Rowan-enabled agent would fare on drug discovery tasks, so we ran a subset of the DrugDiscoveryBench benchmark with and without Rowan tools.

In this post, we'll share how we ran these experiments, how the use of Rowan affected benchmark results, and what this benchmark demonstrates about integrating tools into scientific agents.

Experiment Design

We ran our tests using OpenAI's GPT-5.5-medium because of the high performance per token reported for this model in the original benchmark:

Instead of running the entire 82-task benchmark, most of which is irrelevant to Rowan, we ran a smaller selection of 15 Rowan-relevant tasks: 7 tasks from the SAR/affinity-ranking subset, 7 from the structural-reasoning subset, and the single task from the cheminformatics subset.

We ran our benchmark following the protocol specified by the DrugDiscoveryBench team. Briefly: for each run, the agent was asked a research question and had up to 2 hours to provide an answer. Each run was executed within an isolated Docker container provided by the DrugDiscoveryBench authors, with Codex as the agent harness. We ran the experiment in 2 configurations: "Rowan," which had access to Rowan tools and our skill, and "baseline," which did not. We ran each configuration on each research question three times to control for stochastic variation.

Each rubric had a set of weighted criteria for what a correct answer should contain, and the GPT-5.4 judge marked each one as met or not met. The final outcome score was the sum of the weights of met criteria as a fraction of the total positive weights available. Numeric answers counted as correct within about 0.1 percent of the reference value, while identifiers, integer counts, and categorical answers had to match exactly. The rubrics also included negative penalties for specific mistakes, but those were triggered only on the content-moderated Target #7 and Target #14 tasks, so every other task was scored on its positive criteria alone.

Results

Overall, using Rowan led to better benchmark results: Rowan-enabled runs had a mean score of 53.9% while baseline runs had a mean score of 45.7%. These overall metrics obscure a lot of interesting run-by-run variation, though: sometimes the Rowan skill helped because the agent ran calculations and got the right answer, but other times the agent didn't run calculations and performance still varied either stochastically or due to the skill priming the model context for the given task. We'll briefly step through each of the subsets at a high level here—if you just want our high-level takeaways, feel free to skim this and move on to the conclusions.

(Note that we're blinding the targets here to prevent future benchmark hacking.)

SAR/Affinity Ranking

In the SAR/affinity-ranking tasks, the agent was given a crystallographic protein structure and a set of ligands and was asked to rank them based on binding affinity. It's worth noting that the Target #7 task could not be completed owing to prompt refusal, so only 6/7 tasks can be evaluated here.

Although Rowan has many workflows that might be useful for affinity prediction, like analogue docking, molecular dynamics, or FEP (which could run within the time limit using "fast" settings), Rowan calculations were only invoked for 2 of the 6 tasks. In the other 4 tasks, agents tried to run docking locally, reasoned directly from the protein–ligand structures, or searched for reference affinities from ChEMBL. We noticed that Rowan trials were more likely to reference affinities from ChEMBL, even when noisy or inconsistent values, which may be attributable to our skill's emphasis on the importance of experimental binding-affinity values.

Rowan calculations were successfully run for two targets, both of which led to interesting behavior.

Target #3

For the first target, all of the baseline runs used a combined approach of retrieving affinity values from ChEMBL and running local Vina docking to place the ligand without a ChEMBL record. This approach consistently failed.

Two of the Rowan runs invoked the analogue-docking workflow, which did produce the correct ranking for the missing ligand. However, due to some ligands failing Rowan's automatic PoseBusters check, the agents opted to fall back to local Vina and thus reproduced the incorrect ranking.

One Rowan run used the batch-docking workflow, decided it had waited too long (despite the run completing in under 2.5 minutes), and ranked the ligands with local docking without viewing the batch-docking results. (We have time-estimation functionality for workflows, but it was not used here.)

Thus, despite Rowan having a workflow that reliably gave the right answer, no run was successfully able to use this workflow.

Target #4

For the second target, 3/3 baseline runs and 2/3 Rowan runs retrieved known affinity values from ChEMBL and directly reasoned on the provided structures to rank the final ligand. This approach was inconsistent, producing the correct ranking four times and an incorrect ranking once.

One Rowan trial ran analogue docking and batch docking to rank the final ligand, decided that the magnitudes of the analogue-docking results were too large and discarded them, and used the final rankings from batch docking (which were correct). It's remarkable that this is the only clear-cut "win" for simulated affinity rankings among the 18 independent runs here.

Structural Reasoning

For each of these tasks, the agent was provided a crystallographic structure with a bound ligand and asked for a physical property of that ligand as it existed in the pocket (e.g. atomic charge, formula, or overall charge).

A Rowan calculation was only invoked in a single run for a single task, which asked for the net charge of a bound inhibitor. The Rowan trial ran a macroscopic pK_a workflow to correctly determine the protonation state of the compound and cross-validated the results with Open Babel. Every other run failed, relying solely on data retrieval for this task, which is a nice validation of the tool-based approach.

The Rowan-enabled runs that did not invoke Rowan workflows generally varied a lot in performance compared to baseline runs, likely due to stochastic variation. Despite the Rowan macroscopic- and microscopic-pK_a workflows being great choices for net charge and per-atom charge prediction, the trials preferred to invoke Open Babel tools locally for computational modeling. This generally failed or produced incorrect results.

Cheminformatics

This task provides a drug's structure and asks for its chemical formula in the major form at a specific pH. Every baseline trial used Open Babel's rule-based pH model, which produced an incorrect prediction in all three cases. Only one of these trials mistrusted the results and achieved the correct answer through a combined web search and direct reasoning approach.

All three Rowan trials used the macroscopic pK_a workflow to find the microstate that dominated at pH 4 and correctly reported the chemical formula of that microstate. This is pretty much a best-case scenario for Rowan—we have a workflow that directly addresses this question—so it's good to see that adding the tool works here.

Conclusions

Here are some of our big takeaways from this benchmark:

• Discoverability is the biggest gap. In many cases, there were Rowan workflows that would have been appropriate, but the agents preferred direct reasoning or local modeling. Improving the skill's guidance on when to use our workflows is our biggest opportunity for improvement—even the best simulation in the world can't drive real-world value if it never gets run. (This result was very surprising to us.)
• Rowan calculations mostly improved performance. When Rowan calculations were actually run, they provided correct answers more often than local calculations. Many Open Babel and local Vina runs yielded incorrect results that led the agent astray, while incorrect (and sometimes correct) Rowan results were typically identified as faulty by the agent and discarded. While there are plenty of ways that Rowan or any other in silico model can go astray, Rowan did seem generally helpful here.
• Rowan calculations gave consistent results, while direct reasoning didn't. GPT-5.5-medium's preferred method for completing tasks was to directly reason on the provided structures. While its capabilities were impressive on occasion, the results were inconsistent and irreproducible even under identical starting conditions—which is hardly comforting for scientists trying to figure out how to make important real-world decisions. In contrast, simulation tools run through Rowan give reproducible results, providing a way to diminish the chaos of unstructured reasoning.

Zooming out, running these experiments reminds us that building effective scientific AI systems is, above all, a system-design problem. Even under seemingly ideal conditions—a clean and well-defined scientific problem, simulation tools that are well-matched to the problem and that give the correct answer, and an intelligent AI system capable of autonomously reasoning and using tools—it's still surprisingly difficult to get everything to seamlessly work together to get the right answer for the right reason. Getting this right will require close integration across the stack, not just isolated improvement of any one component.

This benchmark only touches a fraction of Rowan's capabilities. We look forward to seeing how Rowan-enabled agents perform on tasks that require molecular modeling, generative design with property optimization, or even complex multi-step inverse design problems. If you'd like to enable your agents with Rowan, download our skill and use our rowan-python SDK to get started—and if you're interested in talking to our team to collaborate on a benchmark or real-world application, please contact us!