Simulation Tools Improve Agent Problem-Solving

by Corin Wagen · Jul 31, 2026

Agentic AI systems are now good at scientific reasoning and analysis tasks, which in turn is leading to their use in increasingly complex scientific capacities: literature review, molecular design, reaction optimization, and data analysis. Recent work from Anthropic explored how Claude could be used to assist in the interpretation of NMR spectra, one of the most ubiquitous analytical methods in the organic chemistry lab. In their words, "a general-purpose model can be competitive with NMR software and even make 1D inverse elucidation tractable."

The Anthropic work distinguishes between "forward prediction," which predicts the NMR spectrum from a given molecule, and the harder "inverse prediction" problem, which asks the model to figure out which molecule generated the given spectrum. While the forward-prediction problem is solved routinely in academia and industry by a variety of in silico tools, including quantum chemical methods and faster surrogate models like ChemDraw's ChemNMR, MNova's NMRPredict, and CASCADE, the inverse-prediction problem is typically performed by human scientists, who use their chemical intuition and accumulated expertise to enumerate potential structures and test each one. Anthropic found that Opus 4.7 could directly solve inverse-prediction problems for simple molecules, while more complex molecules required additional structural priors.

We recently released an NMR workflow which solves the forward-prediction problem using an equivariant graph transformer ("MagNET," developed with collaborators at Merck, MIT, Harvard, &c—see the preprint on ChemRxiv). We're generally interested in understanding how and when to integrate simulation tools with AI agents, and we hypothesized that giving an AI agent access to our forward-prediction NMR tool might improve agent performance on inverse-prediction problems. There are intuitive reasons why this might be true: use of an NMR simulation tool acts as an external check on agentic reasoning and allows for iterative experimentation, ideally allowing the model reason coherently for longer. On the other hand, it's possible that imperfect simulation tools are simply distracting to AI agents, consuming context & reasoning tokens without contributing any insight.

In this post, we report the results of our initial benchmarking in this area. Our results support our hypothesis: external simulation tools do noticeably improve agent performance, allowing models to think for longer and solve problems that they can't solve otherwise.

Design

We wanted to focus our benchmark on difficult structural elucidations that would require extended reasoning to solve correctly. To identify molecules with potentially challenging spectral assignments, we looked for cases where a molecule's structure had been misreported and later corrected with the aid of ¹³C NMR computations. We compiled 8 such cases for this benchmark from Novitsky and Kutateladze's 2022 work. While this work was released before the model's training cutoff, the agents' performance makes us relatively unconcerned about data leakage, with some exceptions (Appendix 3).

NMR shieldings were predicted using the aforementioned MagNET model, which was trained on millions of DFT-computed isotropic chemical shifts. Like many in silico models of experimental data, MagNET is predictive but imperfect: on average, MagNET-based simulation of the true structures gave predicted shifts with an MAE of 1.20 ppm and an RMSD of 1.60 ppm relative to reported experimental values.

Each agent was given the chemical formula, the experimental ¹³C chemical shifts, and the ability to predict ¹³C NMR shifts up to 10 times via a simple MCP integration with Rowan. Web search and other tools were disabled for this evaluation. We kept the prompt simple to give the agents room for creativity—here's the prompt for the first question:

Prompt:

Solve this closed-book 13C NMR structure-elucidation problem.

Molecular formula: C10H12OS
Solvent: CDCl3
Experimental 13C chemical shifts, δ ppm, unordered and with duplicates
preserved: [158.5,136.1,128.7,128.2,128.2,125.5,125.5,85.0,72.0,15.1]

You must obtain at least one completed simulate_13c(smiles) result for a valid,
formula-matching candidate. You may invoke simulate_13c at most 10 times total.
Every invocation consumes one call, including invalid or formula-mismatched
candidates.

Use the simulation results to compare and refine plausible structures before
answering. Lower MAE indicates better agreement with the experimental 13C NMR
data and makes a candidate more likely to be correct; high MAE is evidence
against a candidate.

Return exactly:
{"final_smiles": "..."}

It's possible to follow the agents' problem-solving process by watching which simulations they chose to run. Here's a sample successful set of simulations for the above prompt: the agent correctly identifies that there's an alkene, a benzene ring, an ether, and a thioether right away, but has to try a few different ways of combining the functional groups before figuring out the correct answer.

We ran 7 replicates per molecule across 8 molecules for a total of 56 runs per model. We also ran all of the models without any external tool calls, removing that section of the prompt, to understand the effect of giving the model the tool. After a few early investigations with different models (Appendix 2), we opted to use GPT-5.6 Sol for the bulk of this study.

A run was counted as successful only if the agent supplied the correct SMILES, modulo canonicalization, of the product (or its enantiomer). To track partial successes, we also (1) recorded the percentage of answers with correct connectivity but incorrect stereochemistry and (2) tracked the mean Tanimoto similarity (Morgan fingerprint, radius 2) of submissions to the true answer.

Results

We found that use of the external NMR tool led to a dramatic improvement in agent performance. While GPT-5.6 Sol xhigh was only able to solve 21% of the problems on its own, the success rate improved to 52% with the tool integration. Use of increased reasoning seemed to help the tool-use runs, with the best results found at "xhigh" effort (see full data in Appendix 1):

(The absolute number of trials is pretty small here, so we've added the actual number in parentheses beneath the percentage score. This post is statistically underpowered and as such results should be taken as preliminary.)

Since scoring exact structural success penalizes near-miss guesses (e.g. incorrect stereochemistry) the same as totally incorrect answers, we also compared the average Tanimoto similarity of all guesses. Once again, tool use generally led to improved performance:

(It's worth noting that, as in the above example, even "wrong" answers have substantial Tanimoto similarity with the correct answer. The agents are in the right ballpark even when they're getting things wrong, with or without Rowan.)

Use of the external simulation tool allowed the model to think effectively for longer. While increasing the effort level without tool use led to minimal improvement in performance, models with tool integration were able to think longer and use the tools more: the average number of NMR tool calls increased from 4.9 on "low" effort to 6.8 on "xhigh" effort. This translated into better performance and higher total token usage:

(Model cost was chosen to combine input/output/cached tokens into a single metric and uses current OpenAI pricing.)

Not all questions were equally difficult. Some structures proved relatively simple to assign; for instance, case #5 was correctly assigned in 51/56 independent runs, matching Anthropic's finding that simple arenes can be solved without external tools. In contrast, case #6 was solved mainly by GPT-5.6 Sol xhigh with Rowan (3/7 runs, 2 non-xhigh solutions), while case #2 remained unsolved across all reasoning levels. A full record of success rates per question can be found in Appendix 3.

Viewing the step-by-step tool calls for case #6 shows a process of iterative experimentation; in one run, GPT-5.6 Sol quickly guesses the diepoxide cyclohexene scaffold but has to figure out relative stereochemistry and where the methyl group goes.

In contrast, other runs got confused and explored incorrect regions of chemical space. One particularly adventurous agent at the high effort level examined a wide variety of scaffolds:

Discussion

This benchmark demonstrates that external simulation tools can help agents solve certain classes of challenging scientific problems. When given access to the proper tools, agents are able to reason for longer and perform significantly better on the benchmark than agents without tools; naively attempting to increase agent thinking without tools does not produce the same results.

While we haven't rigorously assessed human performance on these problems, we suspect that the combination of GPT–5.6 Sol xhigh and Rowan already achieves superhuman performance on this class of problems. Incorrect structural identification based on ¹³C NMR is commonplace in the literature (see e.g. this recent JACS paper), and we suspect the median PhD-level organic chemist would be hard-pressed to answer many of these questions from the prompt above.

More philosophically, this study is a convenient model system for the intersection of agents and tools. The answer is rigorously verifiable (unlike many open-ended multiparameter optimizations), the tool is relevant to the problem but inadequate on its own, and correct problem-solving requires relatively creative and open-ended thinking. We expect that further improvements to interfaces, harnesses, prompts, and models can all be devised that give increased performance on scientific tool-use tasks, and simple benchmarks like these may prove to be convenient model systems.

Rowan isn't mainly an LLM benchmarking company, and there are some obvious next steps from this work that we've left undone:

• Re-run this analysis with a larger dataset of unpublished molecules, more replicates, and different models.
• Incorporate additional physical information—physical properties, ¹H NMR spectra, IR spectra, procedure used to generate the compound, and so on—and relevant simulation tools.
• Include additional tools, like literature search and a Python sandbox.
• Study modifications to the prompt or harness with the goal of improving overall performance.

Analogous tool-use experiments could be designed for many other problems in chemistry, drug discovery, and materials science. We continue to be interested in the intersection of tool use & scientific agents and would be happy to discuss continued research, benchmarking, & deployment in this space.

Thanks to Ari Wagen, Derek Alia, Eli Mann, Ishaan Ganti, and Kat Yenko for helpful discussions.

Appendix 1: Full GPT-5.6 Sol Results

Appendix 2: Comparison of Different Models

We examined models other than GPT-5.6 Sol for this task (with tool use). Here's what we found:

• Use of the smaller GPT-5.6 Terra led to generally poor performance even with tool use on "medium" effort, leading us to switch to the more capable GPT-5.6 Sol.
• Kimi K3 performed well when it returned a structure output; but repeated timeouts and missing structured outputs made a full study impractical.
• Claude Opus 5 performed well but occasionally stalled, hitting the 1-hour timeout repeatedly, making it less operationally reliable than GPT-5.6 Sol (and much more expensive).
• Claude Sonnet 5 was more expensive than GPT-5.6 Sol while performing significantly worse; unlike Opus, however, it did consistently finish.
• Grok 4.5 and Muse Spark 1.1 were the cheapest models; Grok 4.5 had accuracy above GPT-5.6 Terra and comparable to Claude Sonnet 5, while Muse Spark 1.1 was slightly better.

These results are unoptimized and should not be taken as a rigorous comparison of models; it's likely that different prompts or harnesses could lead to different outcomes here.

* Reported aggregate values treat failed attempts as incorrect.

Appendix 3: Successes By Question

All cases were given 7 attempts per model, so a score of 7 indicates that all attempts were successful.

The two successes on case #8 are surprising and prompted us to rerun this case several times with a modified prompt which asked the model for an explanation of its reasoning. In the cases where GPT-5.6 Sol again guessed the structure correctly, it gave explanations like this (emphasis and formatting added):

C₁₉H₂₀O₃ requires 10 degrees of unsaturation. The 219.9-ppm ketone and 172.2/51.9-ppm methyl ester identify both carbonyl functions. Six aromatic carbons and the 148.8/114.0-ppm alkene, together with the remaining unsaturation count, require a compact tetracyclic framework. Two methyl signals and the congested 39–55-ppm bridgehead/methylene region support revised ipomone. The constitution is strongly supported, but ¹³C data alone cannot securely establish relative or absolute stereochemistry.

This indicates that the model gets the right answer by gaining enough structural data to match this molecule to the structure of ipomone, which is impressive since ipomone is a quite obscure natural product first reported in 2021. Future benchmarking should focus on unpublished data to prevent leakage like this (whoops).

Appendix 4: Full SMILES and Shieldings