PHREEQC-MCQ-200: A Diagnostic Benchmark for Tool-Augmented Scientific Simulator Agents

PHREEQC-MCQ-200 is a diagnostic benchmark for tool-augmented agents using a deterministic scientific simulator. Beyond aggregate accuracy, we measure retention—whether tool use preserves answers a model already gets right—and show that overall gains can coexist with item-level regressions. We also compare raw-output access with a metadata-guided TOC interface, finding that the effectiveness of output access depends on model capability.

Paper: arXiv:2607.00436


Poster

When Tools Help and Hurt Scientific Simulator Agents: A PHREEQC-MCQ-200 Case Study

PHREEQC-MCQ-200 poster

Click the poster to open the full-resolution PDF.


← Back to main website