A recurring mistake in agent evaluation harnesses is collapsing two very different questions into one gate: "did the agent's tool call execute" and "was the agent's final answer good." They fail independently, and conflating them makes regressions unreadable.
Two gates, not one
A verifier stage checks mechanical correctness: did the tool call resolve, did the output match the expected schema, did the run terminate instead of looping. It's deterministic and cheap, and it should fail fast before anything expensive happens downstream.
A judge stage is a separate LLM call (routed through LiteLLM with a fallback provider) that scores the content of a passing run against a rubric: correctness, instruction-following, tool-use appropriateness. It only sees runs the verifier already passed.
Why the split matters for regressions
When a benchmark run fails, the first question is always "which stage failed, and why." If verifier and judge are one blended score, a regression in tool-call formatting and a regression in reasoning quality look identical in the dashboard. Splitting the gates means the regression report tells you which one moved, and LangSmith traces give you node-level replay for the specific stage instead of the whole run.
The pass/fail contract
Each stage emits a structured pass/fail plus evidence, not just a score. That evidence, the actual tool call payload, the judge's rationale, is what turns a red dashboard cell into a five-minute debug session instead of a re-run-and-hope.