Observability for conversational agents

Observability provides insights into your conversational agent's performance, user satisfaction, and areas for improvement. Use these tools to monitor production behavior and iterate on your agent's design.

Dashboard

Instance Management provides a dashboard view for monitoring your conversational agent's key metrics and performance indicators.

Accessing the dashboard

1. Go to the UiPath Automation Cloud portal.
2. Navigate to Agents > Deployed Agents.
3. Select your conversational agent.
4. Select the Dashboard tab.

Available metrics

The dashboard provides visibility into:

• Usage metrics: Conversation volume and user engagement.
• Performance indicators: Tool usage and active users.
• User feedback: Summary of thumbs up/down ratings.

Note:

Consumption metrics are currently being updated and will be available in an upcoming release.

User feedback

Feedback from users helps identify areas where your agent performs well and where it needs improvement.

How feedback works

1. Users provide thumbs up or thumbs down after agent responses.
2. Users can optionally add comments explaining their feedback.
3. Feedback is attached to the trace and can be reviewed in Instance Management.

Reviewing feedback

Access user feedback through Instance Management:

1. Navigate to Agents > Deployed Agents.
2. Select your conversational agent.
3. Go to the Feedback menu.
4. Review feedback entries with ratings and comments from users.

Use this feedback to identify areas for improvement and understand user satisfaction with agent responses.

Trace logs

Trace logs provide detailed records of agent execution, enabling you to debug issues and understand agent behavior.

Accessing traces

1. In Instance Management, select your agent.
2. Navigate to the Runtime section.
3. Select a conversation to view its trace.

What traces show

Each trace includes:

• LLM calls: Prompts sent to the model and responses received.
• Tool invocations: Which tools were called, with inputs and outputs.
• Timing information: Duration of each step.
• Token usage: Tokens consumed per call.

Using traces for debugging

When investigating issues:

1. Find the conversation in Instance Management or Orchestrator jobs.
2. Open the trace to see the full execution path.
3. Identify where the agent deviated from expected behavior.
4. Use insights to refine your system prompt or tool configuration.

Iterating on your agent

Observability completes the feedback loop in the agent lifecycle. Use insights from production to improve your agent's design.

The iteration process

1. Identify issues: Review dashboards, feedback, and traces for problems.
2. Diagnose root causes: Use traces to understand why issues occur.
3. Update design: Modify system prompts, tools, or configurations.
4. Test changes: Create evaluation tests covering the identified issues.
5. Deploy updates: Publish the improved agent.
6. Monitor results: Verify improvements through observability.

Common improvements based on observability

Observation	Potential improvement
Negative feedback on specific topics	Add or improve Context Grounding indexes
Tool failures	Review tool configuration and error handling
Long response times	Optimize tool selection or switch models
Users asking for unavailable features	Update system prompt to set expectations

Audit and compliance

AI Trust Layer Audit

The AI Trust Layer provides a complete audit trail of LLM calls and agent behavior for compliance and governance purposes.

Access AITL audit logs through the Admin portal for:

• Complete history of model interactions
• Inputs and outputs
• Data residency and compliance verification

For detailed deployment options, refer to viewing audit logs

Next steps

• Design: Update your agent based on insights
• Evaluation: Create tests from production scenarios
• Agent traces: Detailed trace documentation
• Agent score: Understanding agent health scores