评估智能体

在本地运行评估追踪，并审核进入 Studio Web 的结果。

Step 8 - Create evaluation tests

Evaluations test how well your agent performs across a range of inputs, including whether it calls your new tool at the right moments. The uipath-agents skill includes the complete evaluation framework reference: evaluator types, eval set schema, directory structure conventions, and best practices like using gpt-4.1 (not mini) for LLM judge evaluators. Your coding agent uses this to produce correct evaluator configs and test sets from a short prompt.

询问您的编码智能体：

Create an evaluation set for the intake classifier agent with 5 test cases:

1. A clearly trivial request (e.g., deliver a letter) - no creature named, get_challenge_rating should not be called
2. A standard request (e.g., escort a caravan) - no creature named, get_challenge_rating should not be called
3. A heroic request naming a goblin (e.g., clear a goblin stronghold) - get_challenge_rating should be called exactly once, querying for a goblin, and no other creature
4. A legendary request naming a dragon (e.g., slay a dragon) - get_challenge_rating should be called exactly once, querying for a dragon, and no other creature
5. An edge case that's ambiguous on difficulty but also names no specific creature - get_challenge_rating should not be called; this tests that the agent doesn't over-call the tool just because a case is hard to classify

Use both a semantic similarity evaluator (to check the output) and a trajectory evaluator (to check whether get_challenge_rating was called, and with what search term, matching the expectations above).

Include evaluator config files in evaluations/evaluators/ and the eval set, named smoke-test.json, in evaluations/eval-sets/. Use gpt-4.1-2025-04-14 as the model in the evaluator configs. Each evaluator config must include a populated defaultEvaluationCriteria - use {"expectedOutput": {}} for the semantic evaluator and {"expectedAgentBehavior": ""} for the trajectory evaluator. Empty {} fails schema validation.

---

Step 9 - Run evaluations

在本地运行评估集：

uip codedagent eval agent evaluations/eval-sets/smoke-test.json --workers 3 --output-file eval-results.json

评估框架通过智能体运行每个测试用例，并对结果进行评分。

分数	它测量什么
语义相似度	智能体输出与预期输出的匹配程度
智能体轨迹	Whether the agent called get_challenge_rating when (and only when) it should have

备注：

Trajectory now means something here. With the tool in place, expect trajectory scores close to 1.0 across all five cases: no tool call on the trivial, standard, and ambiguous cases, and exactly one correctly-targeted tool call on the goblin and dragon cases. A low score tells you the agent called the tool when it should not have, skipped a call it should have made, or looked up the wrong creature, not just whether the final tier happens to be right.

For semantic similarity, scores above 0.8 are generally solid; expect the same for trajectory now that it is tracking something specific. Review eval-results.json to see how your agent performed.

在下一步中连接到 Studio Web 后，从 CLI 运行uip codedagent eval run会自动将结果上传到 Studio Web；评估集和评估集将显示在运行下的评估集选项卡中。

备注：

Studio Web 运行评估按钮不是一回事情。该按钮会触发需要 Python 运行时支持的 Cloud Robot 执行，这是一个更复杂的设置，不属于本实验室的范围。改为使用 CLI 中的uip codedagent eval run ；无论哪种方式，结果都会在 Studio Web 中显示。

---

确认本地评估结果后，在下一部分中，您即可准备将项目连接到 Studio Web。