Evaluate agent trajectory

Rerun the existing evaluation set on the tool-using quest classifier, then build a tool-aware set scored by semantic similarity and trajectory evaluators.

With the agent calling its tool only when it should, you can evaluate it.

Step 6 - Rerun your existing evaluation set

Your agent already has an evaluation set: the five cases from the low-code lab. Before you build anything new, rerun it to check what the change did to behavior you had already accepted.

1. In the project explorer, open the agent's Evaluation Sets node, and find the set from the low-code lab.
2. Select Run for the evaluation set.
3. When the run finishes, select View all results, then select the most recent run.

Expect the returned tiers to hold, and expect the semantic similarity scores to barely move. For example, the rats and dragon cases can drop from 100% to 98%.

Their saved outputs come from the agent before it had a tool, so the expected reasoning does not cite a challenge rating and the new reasoning does. The meaning is close enough that semantic similarity hardly registers the change. The exception is heroic-relic, which can score lower: its new reasoning cites a guard's challenge rating and drops details the saved answer included, such as possible magical defenses.

The agent now reaches its answer a different way: it calls a tool, then sets the tier from the challenge rating it gets back. A semantic similarity score compares answers, so it can't see that change. To check how the agent got to its answer, you need a trajectory evaluator, which the next step adds in a new set that describes the new behavior.

Wichtig:

Evaluation scores are non-deterministic. The evaluator is itself a language model, so scores vary between runs. A two-point drop like this one is within that normal variation.

---

Step 7 - Build a tool-aware evaluation set

The new set checks two things: what the agent answered, and how it got there.

• Semantic similarity - compares the agent's output to the expected output by meaning, not exact wording.
• Trajectory - scores how the agent reached its answer from its tool calls: whether it called the tool, what it searched for, and in what order. It tells you whether the tool description and system prompt are doing their job.

Capture the remaining runs

You already have three runs from Step 5. In the Debug panel, run the agent on two more quests:

Escort a merchant caravan through the hill country to the coast.

Recover a stolen relic from a noble's estate. The guards are many, the owner is powerful, and nobody who has tried has come back.

Neither quest names a monster to look up. The relic quest is the harder test: the guards are people the party may have to fight, so one lookup for guards is reasonable, but the relic and its owner aren't threats to search for. An agent that over-uses its tool searches for those too.

Create a new evaluation set and add the debug runs

1. In Evaluation Sets, create a new evaluation set and name it Quest Classifier with Tools.
2. Return to the agent's Definition to access the agent's trace history.
3. Open the History tab in the bottom panel, and select the trace for the run that you want to add.
4. Select Add to evaluation set and choose the new set from the drop-down.
5. In the Add Trace dialog, review the inputs and outputs and select Save to save the trace as an evaluation.
6. Repeat this process for the other four runs from Steps 5 and 7 - adding each of them to the same set.

Wichtig:

Save only runs you agree with. The evaluation set treats each saved output as the correct answer. Check that every run reached the expected tier and called the tool only when it should have.

Describe the expected behavior

The trajectory evaluator scores each run against a written description of what the agent should do. Give each case a name and an expected behavior from this table:

Name	Erwartetes Verhalten
legendary-ancient-dragon	Calls Monster Query before answering, with a search term naming the ancient red dragon rather than a general term such as "dragon".
trivial-rats	Calls Monster Query before answering, with a search term for rats, such as "rat", "rats", or "swarm of rats".
trivial-letter	The tool-call history should be empty. No creature is named, so the agent should not call Monster Query, and an empty history is the correct result.
standard-caravan	The tool-call history should be empty. No creature is named, so the agent should not call Monster Query, and an empty history is the correct result.
no-creature-relic	Calls Monster Query only to look up the guards, or does not look up anything at all. Does not search for the relic or its owner.

Hinweis:

Describe tool use, not the answer. The trajectory evaluator reads only the agent's tool-call history and never sees the final answer, so describe whether the agent calls a tool and what it searches for, and allow reasonable variations in search terms. When the agent shouldn't call a tool, say that an empty history is the correct result, or the evaluator reads it as missing evidence.

1. Open Evaluation Sets and select View details to open the new evaluation set you created.
2. For each case, select Edit to open its settings, and then:
   1. Rename the case to its name in the table.
   2. Select the Trajectory checkbox, if it is not already selected.
   3. Under Agent tools, leave Simulate off for Monster Query, so the tool makes real calls. The trajectory evaluator is less useful when tool calls are mocked.
   4. Paste the expected behavior into the Behavior and output notes field.
   5. Wählen Sie Speichern.

Attach both evaluators and run the set

1. In the new set, select the pencil icon next to Applied Evaluators.
2. Check that the list includes both a semantic similarity evaluator and a trajectory evaluator. If the project has no trajectory evaluator yet, create one from the same dialog.
3. Select Evaluate set to run all of the evaluations in the list.
4. When the run finishes, select the most recent run. Each case shows a semantic similarity score and a trajectory score, from 0 to 100.
5. Select any case to see the run, its scores, and each evaluator's explanation.

Wichtig:

A missing evaluator fails silently. If you forget to attach the trajectory evaluator, the set still runs and shows only semantic similarity scores. If you see one score per case, revisit Applied Evaluators.

Expect high scores from both evaluators with these descriptions. In the run in the preceding screenshot, semantic similarity averaged 98.6% and trajectory averaged 99.4%.

Run the set more than once before you trust a number: the trajectory evaluator is also a language model, and in testing, the same case with the same description scored 45% on one run and 75% on the next. To see how the evaluator scored a case, open it and read its justification.

Hinweis:

You may see a justification mention a redundant first call that returned empty. The evaluator's view of the run lists each tool call twice, even though the agent called the tool once (check the Execution Trail to confirm). Until that's fixed, avoid expected behaviors that limit the number of tool calls.

A low trajectory score with a high semantic similarity score means the agent reached a reasonable answer, but not the way you asked. Typical causes are a search term that is too general (the dragon problem from Step 3), or a tool call on a quest that names no creature. When this situation occurs:

1. Fix the tool description, the system prompt, or the expected behavior.
2. Rerun the set.
3. Compare the results with the previous run.

---

Ergebnisse

Your quest classifier calls Monster Query only when a quest names a creature, cites the challenge rating the tool returns, and has a tool-aware evaluation set that scores both its answers and its tool use.

Wie geht es weiter?

• Building a LangGraph agent - build this same quest classifier, with the same System Reference Document (SRD) lookup tool, as a coded agent in Python.
• Agent evaluations - reference for evaluation sets, evaluator types, and simulated tool calls.
• Open5e API docs - explore the full monster search API and the other fields it can return.
• UiPath Community – Foren, Anleitungen und Entwicklerdiskussionen.