How to Build a Useful Eval Set for a Tool-Using Agent

🧠 Why final-answer evals break down for tool-using systems

Final-answer checks work reasonably well for pure generation tasks because the output is the product. A summary is judged as a summary. A rewrite is judged as a rewrite. But an agent that uses tools is a workflow system wrapped in language. The answer is only the visible tip. Underneath it is a chain of decisions that may be correct, sloppy, unsafe, or wasteful. That hidden chain is often where the real regression lives.

That becomes even more important when you are working with modern LLM agent architecture, where routing, memory, tool use, and orchestration interact. In practice, agent reliability depends less on eloquent wording and more on whether the system chooses the right action under uncertainty.

OpenAI’s evaluation guidance separates failure surfaces like instruction following, tool selection, argument precision, and final task completion because they are not interchangeable. A tool-using agent can fail one and pass another. For example, a customer asks, “Where is my order?” The agent should first ask for an order ID. If it guesses, calls the lookup tool with a fabricated ID, gets an error, then replies with a generic status message, the final text might still sound plausible. The behavior was still wrong. It acted before it had enough information.

This distinction matters in production because wrong actions are expensive in ways that text quality metrics do not capture. Extra tool calls increase latency and API cost. Bad arguments can create downstream failures that look like flaky infrastructure. Premature actions can trigger state changes before a user has actually authorized them. If your eval only says “answer looked acceptable,” you are training your team to miss the exact problems that will be painful later.

LangChain’s trajectory-oriented eval framing makes the same point from another angle: when agents change, the execution path should be evaluated as a first-class artifact, not as a side note. That is what turns evals from QA theater into an engineering control system. You stop asking, “Does this seem okay?” and start asking, “Did the agent behave correctly under this scenario?”

A useful way to think about this is to compare a tool-using agent with a small backend service. You would never test a payment service by only checking whether the final UI message said “payment complete.” You would inspect whether the right account was charged, whether idempotency worked, whether retries were safe, and whether failures were reported honestly. Agents deserve the same discipline because they now sit in the same operational path. They are making decisions, not just producing prose.

🛠️ Start with three dataset buckets, not one blended pile

A useful eval set should not be a random collection of prompts. It should be structured around the kinds of behavior you need to trust. A strong starting point is three buckets: happy path, ambiguous or underspecified requests, and failure or safety-sensitive cases. This simple split is more important than it looks because it forces you to test different decision modes instead of overfitting to clean examples.

The happy-path bucket proves the baseline workflow actually works. These are cases where the user provides enough information, the correct tool should be called, the arguments are extractable, and the final answer should be clear. If your agent cannot pass these consistently, you are not ready to reason about harder cases. But if your eval set is only happy path, you will build false confidence very quickly.

The ambiguous bucket is where many real systems fail. These examples test whether the agent asks clarifying questions instead of guessing. OpenAI’s order-status examples are useful here because a request like “Where is my order?” is not complete enough to justify action. The correct behavior may be to ask for the order ID and do nothing else. That is a behavioral success, even though no tool was called and no final status was produced. This is exactly the kind of case final-answer evals handle badly.

The failure and safety-sensitive bucket captures the cases teams usually discover too late: invalid IDs, tool downtime, topic shifts, stale context, no-tool-needed requests, and prompts where acting would be risky. This bucket matters because production is not a benchmark suite. Real users change subjects mid-thread, omit key fields, and phrase things in messy ways. If your eval set does not reflect that, it is not an eval set for production. It is a demo dataset.

There is also a practical reason to keep these buckets separate in reporting. If you blend all rows into one pass rate, a strong happy-path score can hide weak behavior in ambiguous or risky cases. That makes the system look healthier than it is. A routing policy that works on clean inputs but fails whenever the request is underspecified is not a mature policy. Separate buckets make that visible immediately, which is exactly what an eval set is supposed to do.

📦 What each eval row needs to store

Most teams under-specify their eval rows. They store the input and maybe a reference answer, then wonder why failures are hard to interpret. For a tool-using agent, each row should capture expected behavior at multiple layers. At minimum, that means the input conversation, the expected action, the expected tool if any, the expected arguments if any, whether clarification is required, and any forbidden tools or forbidden actions.

Why this matters is simple: you want to separate planning from execution. If a row only stores an ideal answer, you cannot tell whether the failure came from choosing the wrong tool, extracting the wrong field, skipping a required question, or formatting the final response poorly. Once you store these fields explicitly, your eval harness can localize the problem. That saves real debugging time.

A useful row in prose might look like this. The conversation input is: “Where is my order?” The expected action is “ask_for_order_id.” Clarification is required: true. Expected tool: none. Forbidden tools: lookup_order, refund_order, reset_password. Completion requirement: ask for the order ID clearly and do not imply the order was checked. That one row already gives you enough structure to reject premature action and hallucinated confidence.

Now compare that with a second row. Input: “Check order 12345.” Expected action: “call_lookup_order.” Expected tool: lookup_order. Expected arguments: order_id equals 12345. Clarification required: false. Forbidden tools: reset_password. Completion requirements: return the delivery status from the tool result, include the estimated delivery date if available, and do not invent a tracking number. That level of specification is not bureaucracy. It is what makes the eval useful.

In practice, a row often needs a bit more context than teams expect. You may want a field for prior conversation turns, a field for mock tool outputs, a field for acceptable alternative phrasings, and a field for severity if the row fails. Severity matters because not all failures deserve the same response. Calling the wrong read-only tool in a support flow is bad, but issuing a destructive action without confirmation is much worse. Once severity is attached to examples, release decisions become grounded in risk instead of gut feeling.

This is also where agent state management starts to matter. If your eval rows do not encode what the agent should remember, forget, or carry across turns, you cannot reliably test AI agent memory or reason about short-term vs long-term memory in agents. A strong dataset row quietly documents the expected state transitions that your architecture depends on.

⚙️ Score in layers so each failure class stays visible

The most practical scoring model is layered. First score the decision. Did the agent ask, wait, or act correctly? Then score argument quality. If it acted, did it pass the fields the tool actually needed? Only after that should you score the final response. This layered structure matters because different failures need different fixes. Wrong tool choice is a planning problem. Malformed arguments are often an extraction or context problem. Weak final messaging is usually a response problem. Blending them into one score hides the signal.

OpenAI’s eval guidance supports this decomposition directly, and it aligns well with the way developers actually debug systems. If the agent chose the right tool but dropped a digit from the order ID, you do not need to rewrite the whole prompt. You need to inspect slot extraction, conversation state, or argument formatting. If the agent answered gracefully after a tool error but should never have called that tool in the first place, the issue is action policy, not language quality.

A weighted rubric can make this operational. For example, 30 percent for correct tool decision, 30 percent for correct arguments, 20 percent for final completion quality, 10 percent for efficiency or no unnecessary tools, and 10 percent for recovery behavior. The exact numbers are not universal. They should reflect business risk. In a destructive workflow, tool choice and argument precision should dominate. In a low-risk support workflow, completion quality may matter more.

The deeper reason this helps is organizational, not just technical. Once each layer has a score, your regressions become discussable. Product, engineering, and applied AI teams can look at the same report and agree on what degraded. Without that, every release review turns into subjective transcript reading, which is slow and surprisingly unreliable.

Layered scoring also prevents one good trait from masking a more important bad trait. An agent can be polite, concise, and grounded in tone while still making the wrong decision. If your scoring system rewards the surface more than the action, you end up optimizing for what is easiest to notice rather than what is most important to control. That usually produces agents that feel fluent in demos and unreliable in production.

For teams building AI coding agents or support agents on top of an agent orchestration layer, this separation is essential. It lets you see whether the issue lives in the planner, the tool wrapper, the memory layer, or the response synthesis step. That is how you improve agent reliability without guessing.

🧪 Use deterministic checks first, then judge the softer parts

Not every part of an agent trace needs an LLM judge. In fact, using a judge for everything is usually lazy system design disguised as sophistication. For tool-using agents, some checks should be deterministic because the expected behavior is unambiguous. Tool name matching, required argument presence, argument schema validity, forbidden tool calls, and whether clarification was required are all strong candidates for deterministic evaluation.

This matters because deterministic checks are cheaper, faster, and easier to trust. If the expected tool is lookup_order and the trace shows refund_order, that is not a nuanced judgment call. It is just wrong. If the row says the agent must not call any tool before receiving an order ID, then a premature tool invocation should fail immediately. You do not need a model to tell you that. Deterministic grading also gives clearer failure messages, which speeds up debugging.

Judge-based scoring still has a place, but it should be reserved for softer dimensions. Was the clarification question natural and specific? Did the recovery message after a tool timeout help the user without sounding misleading? Did the final explanation stay grounded in the tool result? Those are harder to encode with simple rules, and a rubric-based judge can help. LangChain’s eval materials reflect this split between deterministic trajectory checks and LLM-based qualitative grading.

A good practical pattern is a two-stage harness. Stage one runs deterministic checks on the trace. Stage two only evaluates the softer dimensions for traces that passed or partially passed stage one. That keeps eval costs lower and avoids wasting judge calls on obviously broken traces. More importantly, it keeps your team honest. If the structure is wrong, no amount of polished language should rescue the score.

Think of this as test layering, the same way backend teams combine unit tests with integration tests. Deterministic checks behave like precise assertions. They answer whether the system followed hard constraints. Judge-based checks behave more like qualitative review. They answer whether the result was helpful, clear, and sensible. Mixing those into one fuzzy judgment weakens both. Keeping them separate makes the harness easier to trust and easier to maintain.

This approach fits well with Responses API agents and the OpenAI Agents SDK, where traces, tool calls, and intermediate steps are explicit enough to score directly. The practical benefit is simple: the better your instrumentation, the less you need to infer from the final answer.

🧭 Trajectory matching should fit the workflow, not force one exact trace

One subtle mistake in agent eval design is making every example rigidly deterministic. Real workflows are not always that clean. Sometimes tool order matters a lot. Sometimes it barely matters. Sometimes one key tool call must happen, but extra exploratory calls are acceptable. If your eval harness cannot express those differences, you either over-penalize reasonable behavior or allow too much sloppiness.

This is where trajectory matching modes become useful. LangChain documents several patterns, including strict, unordered, subset, and superset matching. Strict matching works when the workflow has hard sequencing requirements, such as authentication before account actions. Unordered matching works when multiple required checks can happen in any order. Superset matching is useful when one core tool call must happen but some additional context-gathering is acceptable. Subset matching is valuable when you want to penalize unnecessary tool calls and keep cost under control.

Why this matters becomes obvious in a support workflow. Suppose a user asks for a simple order status lookup. If the agent calls FAQ search, returns policy lookup, and account profile tools before calling lookup_order, the final answer may still end up correct. But the behavior is wasteful and fragile. A subset-style check catches that by encoding that only a small set of actions is acceptable. That is not being pedantic. It is controlling latency and spend.

The key insight is that your eval set should mirror the action policy your system actually needs. Do not demand exact trajectory identity when the workflow allows flexibility. But do not leave flexibility undefined either. Useful evals are opinionated about what variability is acceptable and what variability is a bug.

Developers often underestimate how much this choice affects iteration speed. If matching is too strict, every harmless implementation change looks like a regression and the team starts ignoring eval failures. If matching is too loose, important policy drift slips through. The right matching mode creates a signal that people keep respecting. That is what makes the eval set useful over time instead of becoming another dashboard no one trusts.

This is also a practical concern in broader agent orchestration systems, including multi-agent systems, supervisor worker agents, and workflows with agent handoffs. If your evaluator assumes one exact route through the system, it may flag healthy coordination as failure. If it allows everything, it will miss real regressions. The matching policy needs to reflect the architecture you actually run.

🔄 Include multi-turn context shifts or your confidence will be fake

Single-turn examples are convenient, but they create dangerous optimism. Many agent failures only appear across turns, especially when the same surface phrasing should lead to different actions depending on prior context. A tool-using agent is not just parsing the latest message. It is interpreting the latest message through thread state. That is exactly where stale-slot reuse, wrong references, and premature actions start to creep in.

Imagine a conversation that starts with a refund issue, then shifts. The user later says, “Where is it now?” What does “it” refer to? The order shipment, the refund, or a support ticket? A brittle agent may reuse the wrong entity from earlier context and call the wrong tool with complete confidence. That kind of mistake is common in production because users naturally speak this way. If your eval set only includes neatly scoped single-turn prompts, you will not see it until real transcripts force the issue.

Topic-switch cases are especially important. OpenAI’s guidance highlights subject changes because they frequently trigger wrong tool selection. The failure is often not that the agent forgot language. It is that state management was loose. It latched onto a previous slot, ignored a new intent, or treated old context as still active. That is why these examples belong in the dataset, not just in ad hoc debugging.

Including multi-turn rows also sharpens your thinking about what should be stored in the eval schema. You may need fields for prior turns, current resolved entities, expected entity reference, or whether old slots must be discarded. This is more work, but it pays off because it tests the agent you actually deployed, not the toy version that only exists in clean demos.

Why this matters operationally is simple. Context bugs are hard to spot by reading only the final answer because the language may still be coherent. The problem is hidden in what the agent believed the conversation was about. Multi-turn rows let you inspect that hidden state indirectly through action choice. That makes them one of the highest-value additions you can make once the single-turn basics are stable.

This is where the difference between short-term vs long-term memory in agents becomes practical rather than theoretical. If your agent can retrieve old information, you also need to test when it should ignore it. Frameworks that support memory, including patterns like LangGraph memory or procedural memory for agents, make this more powerful, but they also increase the failure surface. Eval rows need to reflect that reality.

🏷️ Add metadata and versioning so regressions become searchable

An eval set becomes far more useful when each example carries failure taxonomy metadata. Tags like missing_slot, wrong_tool, malformed_args, unnecessary_tool, premature_completion, no_tool_needed, recovery_after_tool_error, and topic_switch let you break one blended score into meaningful slices. This matters because a global pass rate can improve while one critical failure class gets worse. Without metadata, you may never notice.

LangSmith’s evaluation concepts and dataset management ideas are helpful here because they treat datasets as versioned assets, not static files. That framing is exactly right for agent engineering. Your eval set should evolve as the system evolves. Every time a new production bug appears, you should ask whether its trace can be converted into a durable eval example. Once it is in the dataset, that bug stops being a one-off embarrassment and becomes a permanent regression check.

Versioning matters for another reason: your eval set is part of your release logic. If examples are silently edited in place, trend lines become hard to trust. A cleaner pattern is to publish dataset versions and compare runs against the same version when evaluating prompt, tool, or model changes. Then, when you intentionally add new hard cases, you can explain why pass rates moved. That keeps the signal interpretable.

This is where agent evaluation starts to feel like normal software engineering. You are not chasing vague confidence. You are maintaining a versioned test corpus tied to observed failures. That shift is important because it turns evaluation into an operational system. The team can answer what changed, which failure class regressed, and whether the release should proceed.

Metadata also improves the day-to-day debugging loop. If ten examples fail, the first question is not only how many failed, but what kind of failure they represent. Did they cluster around one tool integration? Did they all involve missing identifiers? Did they appear after a model upgrade? Searchable tags make these patterns visible within minutes. Without them, people read transcript after transcript and try to reconstruct the pattern manually, which is slower and far less reliable.

For developer-facing systems, strong metadata also improves agent evaluation and tracing and supports better agent observability. That matters because once an agent reaches production, the hard part is rarely generating one impressive answer. The hard part is understanding why it behaved the way it did across thousands of runs.

📈 Separate routing tests from full workflow tests

Another practical design choice is to split routing evaluation from full workflow evaluation. LangChain’s complex-agent evaluation examples make this distinction clearly, and it is a powerful one. Routing tests focus on the decision boundary: which tool should be chosen, or should no tool be chosen at all? Full workflow tests assess whether the entire task completed correctly after the decision was made.

This separation matters because it localizes failures quickly. If the routing suite fails, the problem is in planning, intent interpretation, or action policy. If routing passes but the full workflow fails, the issue is more likely in argument extraction, tool integration, tool result handling, or response synthesis. Without this split, your team ends up debugging everything at once, which is slow and often misleading.

Take the order-status pattern. A routing test may simply assert that “Where is my order?” should produce no tool call and a clarification request. Another routing row may assert that “Check order 12345” should choose lookup_order. Then the full workflow test adds the trace and tool result expectations: did the agent pass the correct ID, interpret the returned status correctly, and answer clearly without inventing details? Same topic, different debugging scope.

There is a deeper benefit too. This split helps justify the architecture itself. Anthropic’s guidance on building effective agents repeatedly points back to matching evaluation to the workflow pattern you actually use. If a more complex tool-using workflow does not outperform a simpler pattern on representative routing and completion tests, that is a sign to simplify, not to keep stacking prompts and hoping.

That point matters more than it first appears. Agent systems often grow in complexity because each new edge case gets one more prompt rule, one more helper tool, or one more decision layer. Separate routing and workflow tests give you evidence about whether that complexity is buying anything. If routing accuracy is flat but latency and unnecessary tool use are rising, the eval set is telling you something important: the architecture may be drifting away from the actual problem.

This lens is useful whether you are evaluating simple tool calling agents or more complex long-running agents with retries, checkpoints, and durable state. The more moving parts you introduce, the more important it becomes to isolate routing quality from end-to-end workflow quality.

🚦Turn the eval set into a release gate, not a reporting ritual

The payoff of a good eval set is speed, but only if it changes how releases happen. If your team still upgrades models, edits prompts, or swaps tool layers based on a few manual transcript reads, then the eval set is decorative. It needs to act as a release gate. That means defining thresholds by bucket and by failure class, then deciding in advance what blocks deployment and what triggers manual review.

A practical example looks like this. Happy-path cases may need a near-perfect pass rate because these are your baseline capabilities. Ambiguous cases should strongly favor correct clarification behavior and zero premature tool calls. Safety-sensitive cases may require perfect scores on forbidden actions, even if softer response quality is allowed some variance. You might also set a hard cap on unnecessary tool calls because they directly affect cost and latency. This is where subset matching becomes operational rather than academic.

CI reports should not just say pass or fail. They should bucket regressions by class. If a prompt change improves final response quality but increases malformed_args errors, the report should show that explicitly. If a model upgrade reduces wrong_tool mistakes but makes the agent act too early in ambiguous rows, that should trigger manual review before release. This is much better than reading 50 transcripts and trying to intuit the pattern from memory.

OpenAI’s trace-grading framing is useful here because it makes the trace the unit of analysis, and that is exactly what a release gate should consume. You want failed evals to map back to the exact tool calls and intermediate states that caused them. That makes the iteration loop tighter. Instead of debating whether the system feels better, you can inspect the failed traces and fix the actual behavior that regressed.

The reason this improves velocity is a bit counterintuitive. A strict gate sounds like it would slow releases down, but in practice it does the opposite. Teams move faster when they trust the feedback loop. They can make prompt changes, tool updates, or model swaps without turning every release into a manual review exercise. The gate narrows discussion to concrete failures, and that reduces both uncertainty and rework.

For more advanced systems, this is also where durable execution for agents becomes relevant. If an agent can pause, resume, retry, or survive failures across long workflows, your release gate should test those behaviors explicitly. Otherwise, you are only evaluating the easiest path through the system rather than the one users will actually experience.

🔗 Build the eval set around real traces, then keep sharpening it

The best eval sets do not come from brainstorming in a vacuum. They come from real traces, real failures, and real business risk. Start with a concrete workflow such as order status because it exposes the right surfaces clearly: ask for missing information when needed, choose the correct tool, pass the correct arguments, avoid forbidden actions, and answer from the returned result. Then expand the dataset with ambiguous wording, topic shifts, invalid identifiers, and recovery cases as your system encounters them.

This is also why trace-linked observability matters so much. OpenAI’s material on evals and tracing, plus LangSmith’s trajectory and dataset tooling, all push toward the same operational model: use traces as both debugging evidence and eval fuel. When a production issue happens, do not just patch the prompt and move on. Convert the trace into a labeled eval row, version the dataset, and make sure that exact failure cannot slip through silently again. Useful further reading includes OpenAI’s evaluation best practices, trace grading guide, LangChain’s agent eval docs, trajectory eval guidance, complex agent evaluation examples, and Anthropic’s architecture guidance.

The larger lesson is easy to miss: for agents, correctness is not just about what was said. It is about how the system decided to act. That is why a useful eval set stores behavior, not just answers. Once you internalize that, evaluation stops being a disconnected QA task and becomes part of the agent loop itself.

That is also why the article matters beyond one support example. The same principles apply when you are evaluating AI agents for developers, internal copilots, coding assistants, and orchestration-heavy systems built around tools, memory, and state. As soon as actions matter, traces matter.

And that is the point. A strong eval set does not merely tell you whether the agent looks smart. It tells you whether the agent behaved like a system you can afford to trust.