The Short Answer
Agent monitoring differs significantly from traditional system monitoring. An agent rarely "crashes"; it simply performs less effectively. Therefore, classic metrics like availability and error rates are insufficient. An additional layer is needed to measure decision quality, not just technical functionality.
A practical structure includes three levels: a Trace to explain a single conversation, weekly Trends to detect deterioration, and a Business Outcome metric to justify continued operation. Each level addresses a different question and serves a different audience.
Three Levels of Monitoring
| Level | Question It Answers | Audience | Frequency |
|---|---|---|---|
| Trace | Why did this conversation end this way? | Technical Team & Analyst | On Demand |
| Trend | What is changing for better or worse? | Process Owner & Platform Manager | Weekly |
| Outcome | Does the agent justify its existence? | Management & Sponsor | Monthly & Quarterly |
Level 1: Essential Trace Components
A useful Trace allows you to reconstruct a decision without needing to ask anyone. Seven components are crucial: the prompt as phrased, the identified topic, the actually retrieved sections, the activated actions with their parameters, the result of each action, validation points and their decisions, and the reason for termination.
The often-overlooked component is the retrieved sections. Without it, it's impossible to distinguish between two completely different failures: the agent couldn't find the information, or it found it but used it incorrectly. The first is addressed with content, the second with instructions – and misdiagnosing this has wasted weeks in every organization we've seen.
You also need to link the Trace to a business record: a Case, Order, or Opportunity. Without this link, it's impossible to verify if the conversation ultimately led to a resolution or a repeat inquiry.
Level 2: Trends Worth Tracking
Five metrics suffice for most deployments: completion rate without escalation, repeat inquiry rate within a week, percentage of answers with a valid source, action failure rate, and consumption relative to tasks completed.
Interpretation is done in pairs. High containment with a high repeat inquiry rate is not a success. Consumption increasing while tasks remain stable means the agent is working harder for the same result—an early sign of content degradation.
Automated alerts are configured for relative changes, not absolute values: a jump in escalation rate, a drop in the percentage of valid sources, an increase in action failures against a specific external system.
The connection between these metrics and cost is detailed in Agentforce Cost & TCO.
Level 3: Business Outcome
This is the level that dictates the outcome of the quarterly review. Two numbers: what has changed in our predefined process metrics—handle time, churn, agent contact volume—and what is the cost of a completed task compared to a Baseline.
It's crucial to define these metrics upfront and not change them after seeing results. Changing the definition of “success” mid-flight is the biggest destroyer of management confidence in data, even when the change is justified.
From Analysis to Improvement Backlog
The weekly report is not the deliverable. The deliverable is a work queue. The effective practice is: sample twenty failed or escalated conversations, classify them by root cause—content gap, incorrect tagging, unclear instruction, action failure, or out-of-scope request—and create a work item for only the largest category.
The rule to prevent distraction: address one root cause per week. Organizations that try to fix five concurrently never know in the end what improved the metric and what worsened it.
Content gaps identified here are direct input for the writing backlog – the process is detailed in Knowledge Management for Agentforce.
Scenario: A Quiet Decline Caught in Time
A financial services company deployed an internal agent that was stable for four months. In the fifteenth week, the escalation rate gradually increased without anyone complaining – agents simply took over and completed the handling themselves.
The alert configured for a relative increase in escalations triggered an investigation. Sampling Traces showed that in half of the new cases, no relevant section was retrieved. The reason: a product policy change led to the automatic archiving of eleven articles, and no alternatives were written.
The fix took two days and required no changes to the agent. Without the monitoring layer, the gap would have only been discovered when a manager asked why average handle time had increased—likely a quarter later.
Risks and Mitigation Actions
| Risk | Appearance | Mitigation Action |
|---|---|---|
| Technical monitoring only | Everything is green, but answers are worse | Quality & escalation metrics alongside availability |
| Trace without retrieval content | Months of guessing between content and model | Mandatory logging of retrieved sections |
| Report without work queue | Data presented but nothing changes | Weekly sampling and work item for one root cause |
| Changing definitions mid-way | Loss of confidence in data | Define success metrics upfront |
| No link to business record | Cannot identify repeat inquiries | Link Trace to Case or Order |
Recommended Dashboard
| Metric | Definition | Alert Threshold |
|---|---|---|
| Completion Rate | Resolution without escalation or repeat inquiry | Significant relative week-over-week drop |
| Escalation Rate | Percentage of transfers to a human agent | Sustained relative increase |
| Valid Source | Percentage of answers with existing and valid citation | Drops below a defined threshold |
| Action Failures | Percentage of failed actions by target system | Increase in failures against a specific target |
| Consumption per Task | Consumption units per completed task | Increase without proportional task growth |
When guidance is needed to establish a monitoring layer and weekly improvement process, the Agentforce & AI Service is the practical next step.
Observability Setup Checklist
- ☐ Trace records prompt, topic, retrieved sections, actions, and result
- ☐ Every Trace is linked to a business record
- ☐ Retention policy differentiates between metadata and conversation content
- ☐ Only five to seven trend metrics have been selected
- ☐ Alerts are defined for relative changes, not absolute values
- ☐ A routine for weekly sampling of failed conversations exists
- ☐ A taxonomy of root causes of failure has been defined
- ☐ The business process owner participates in the weekly review
- ☐ Success definitions were established before measurement began