LLM-as-judge and synthetic enterprise environments

Problem

Two problems that turned out to be one problem.

The first: scoring agent behavior at scale. Human review of trajectories is accurate and completely non-viable past a certain volume. Automated scoring is viable and, done naively, measures the wrong thing. A judge model reading a final answer will happily reward an agent that got there by accident, and it will punish a correct answer phrased unusually.

The second: the environments themselves. Early agentic tasks tend to run against fixtures with no world behind them. The agent calls a tool, gets a canned response, and never faces the consequence of having written something wrong. Agents trained in that setting learn that actions are free.

They're one problem because the quality of the judge is bounded by the quality of the environment. If the environment can't be wrong, the judge has nothing to check against.

How I thought about it

I wanted as much of the scoring as possible to be checkable against state rather than against opinion. That means the environment needs to be good enough to hold the truth. So I built the environment side first and let it define what the judge was responsible for.

The split I settled on: programs verify anything that touches state, and the judge handles reasoning quality and tool-use appropriateness, which are genuinely judgment calls. Anywhere the judge is doing work a verifier could do, that's a design failure, not a prompting one.

Architecture

The environment layer is multi-domain synthetic enterprise databases in Snowflake, built with real referential integrity. Records relate to each other the way they would in an operating company, so an agent that writes an inconsistent record produces a database that is actually inconsistent, and the verifier can see it.

The evaluation layer runs in stages. Deterministic verifiers check end state against the task's success condition. Trajectory analysis compares the agent's execution trace against golden trajectories using diff-based comparison, which is how you catch an agent that reached the right answer through a route you don't want to reinforce. LLM-as-judge scoring, driven by rubrics with explicit criteria, handles reasoning quality. Difficulty rubrics tier the output so the reward signal isn't flat across trivial and hard tasks.

Calibration wraps the whole thing. Judge behavior gets checked against a golden set built with human review, and drift in judge scoring is treated as a regression rather than a curiosity. The golden set is versioned and re-run whenever the rubric, the prompt, or the underlying model changes, because all three of those move judge behavior and only one of them announces itself.

What I built

The Snowflake schema design for the synthetic enterprise environments. The LLM-as-judge and reward pipelines that score reasoning and tool use at scale. The task verifiers and difficulty rubrics. The golden datasets and the diff-based trajectory analysis that sits between the two. On the multimodal side, the earlier version of this: multi-turn validation frameworks and Jira-integrated QC question sets, which is where a lot of the process design was worked out before it moved to agentic tasks.