Evaluieren Sie den Agent

Save five debug runs of the quest classifier as a Studio Web evaluation set, name each test case, and score the agent with a semantic similarity evaluator.

With five runs in your history, you can turn them into an evaluation set.

Step 5 - Save your test runs as an evaluation set

An evaluation set is a list of test cases: an input, plus the output you expect. Running the set scores the agent against every case, so you can tell whether a later prompt change broke a tier you had already gotten right.

You build the cases from the runs you just made, so you do not enter any expected outputs by hand.

Save the latest run

1. In the Execution Trail for the last of your five runs (the relic quest), select Add to evaluation set.
2. Review the input and output values in the dialog.
3. Select Save to add the run to the evaluation set.

Wichtig:

Save only runs you agree with. The evaluation set treats the saved output as the correct answer. A wrong tier saved here becomes a test that rewards the wrong behavior.

Add the other runs from history

1. Open the History tab in the bottom panel. It lists every run of the agent.

2. For each of the other four runs from Step 4, select the beaker icon to add it to the same set.

   

Name each test case

Saved runs get generic names. Rename each case so its name says what it tests:

Quest	Name
Clear the rats out of the inn cellar...	trivial-rats
Slay the ancient red dragon terrorizing the countryside.	legendary-dragon
Deliver a sealed letter to the magistrate in the next town.	trivial-letter
Escort a merchant caravan through the hill country to the coast.	standard-caravan
Recover a stolen relic from a noble's estate...	heroic-relic

1. Open the Evaluation Sets node of the solution in the project explorer. You can also reach evaluation sets from the Evaluations tab in the bottom panel.
2. Open the set you saved the runs to. All five cases are listed.
3. Select Edit on a case.
4. Enter the name for that case from the preceding table.
5. Check that the input is the quest text and the expected output is the run's tier and reasoning.
6. Wählen Sie Speichern.
7. Repeat steps 3 to 6 for each remaining case.

---

Step 6 - Run the evaluation set

Run the evaluation set to score the agent against all five test cases with a semantic similarity evaluator.

1. In Evaluation Sets, select View details on your set.

2. Check that Applied Evaluators lists a semantic similarity evaluator, such as Default Evaluator.

3. If Applied Evaluators is empty, select the pencil icon next to it and add a semantic similarity evaluator.

4. Return to Evaluation Sets and select Run on your set.

   

5. When the run finishes, select View all results.

6. Select the most recent run from the list to see the results for each evaluation in that run.

7. Select any case to see its output, its score, and the evaluator's explanation.

Each case gets a score from 0 to 100. The semantic similarity evaluator compares the meaning of the agent's output to the expected output, so a rerun that reaches the same tier with differently worded reasoning still scores well, and a run that lands on the wrong tier scores low.

Wichtig:

Evaluation scores are non-deterministic. The evaluator is itself a language model, so scores vary slightly between runs. Watch for a case that drops sharply.

Studio Web also offers a trajectory evaluator, which scores the steps and tool calls the agent used to reach its answer. On this agent it has few steps to score: the agent answers in a single step with no tool calls. Trajectory becomes the more useful score once an agent calls tools and makes decisions along the way.

From here on, rerun the set after every prompt change. If a case that used to pass starts failing, the change caused a regression.

---

Ergebnisse

Your quest classifier has an evaluation set of five named test cases and a semantic similarity score to compare against after each prompt change.

Wie geht es weiter?

• Adding Tools to Your UiPath Agent - give an agent a live API tool, and evaluate how it uses that tool with a trajectory evaluator.
• Building a LangGraph agent - build this same quest classifier as a coded agent in Python, including a live lookup tool.
• Agent evaluations - reference for evaluation sets, evaluator types, and running evaluations.
• UiPath Community – Foren, Anleitungen und Entwicklerdiskussionen.