Avaliar o agente

Execute traces de avaliação localmente e revise os resultados que fluem para o Studio Web.

Step 8 - Create evaluation tests

Evaluations test how well your agent performs across a range of inputs, including whether it calls your new tool at the right moments. The uipath-agents skill includes the complete evaluation framework reference: evaluator types, eval set schema, directory structure conventions, and best practices like using gpt-4.1 (not mini) for LLM judge evaluators. Your coding agent uses this to produce correct evaluator configs and test sets from a short prompt.

Pergunte ao seu agente de codificação:

Create an evaluation set for the intake classifier agent with 5 test cases:

1. A clearly trivial request (e.g., deliver a letter) - no creature named, get_challenge_rating should not be called
2. A standard request (e.g., escort a caravan) - no creature named, get_challenge_rating should not be called
3. A heroic request naming a goblin (e.g., clear a goblin stronghold) - get_challenge_rating should be called exactly once, querying for a goblin, and no other creature
4. A legendary request naming a dragon (e.g., slay a dragon) - get_challenge_rating should be called exactly once, querying for a dragon, and no other creature
5. An edge case that's ambiguous on difficulty but also names no specific creature - get_challenge_rating should not be called; this tests that the agent doesn't over-call the tool just because a case is hard to classify

Use both a semantic similarity evaluator (to check the output) and a trajectory evaluator (to check whether get_challenge_rating was called, and with what search term, matching the expectations above).

Include evaluator config files in evaluations/evaluators/ and the eval set, named smoke-test.json, in evaluations/eval-sets/. Use gpt-4.1-2025-04-14 as the model in the evaluator configs. Each evaluator config must include a populated defaultEvaluationCriteria - use {"expectedOutput": {}} for the semantic evaluator and {"expectedAgentBehavior": ""} for the trajectory evaluator. Empty {} fails schema validation.

---

Step 9 - Run evaluations

Execute o conjunto de avaliação localmente:

uip codedagent eval agent evaluations/eval-sets/smoke-test.json --workers 3 --output-file eval-results.json

A estrutura de avaliação executa cada caso de teste por meio do seu agente e pontua os resultados.

Pontuação	O que ele mede
Similaridade semântica	O grau de semelhança da saída do agente com a saída esperada
Trajetória do agente	Whether the agent called get_challenge_rating when (and only when) it should have

Observação:

Trajectory now means something here. With the tool in place, expect trajectory scores close to 1.0 across all five cases: no tool call on the trivial, standard, and ambiguous cases, and exactly one correctly-targeted tool call on the goblin and dragon cases. A low score tells you the agent called the tool when it should not have, skipped a call it should have made, or looked up the wrong creature, not just whether the final tier happens to be right.

For semantic similarity, scores above 0.8 are generally solid; expect the same for trajectory now that it is tracking something specific. Review eval-results.json to see how your agent performed.

Depois de se conectar ao Studio Web na próxima etapa, a execução de uip codedagent eval run a partir da CLI carrega os resultados para o Studio Web automaticamente; eles aparecem na aba Conjuntos de avaliação em Execuções.

Observação:

O botão Avaliações do Studio Web Run não é a mesma coisa. Esse botão dispara uma execução de robô na nuvem que requer suporte para runtime do Python — uma configuração mais envolvida fora do escopo deste laboratório. Em vez disso, use uip codedagent eval run a partir da CLI; os resultados aparecem no Studio Web de qualquer maneira.

---

Com os resultados da avaliação local confirmados, você está pronto para conectar o projeto ao Studio Web na próxima seção.