Agentforce Testing: Test Sets, Scorers, & Edge Cases

The Short Answer

Agent testing isn't classic software testing. There's no single correct answer; the same question can be phrased twenty ways, and a "failure" is often a reasonable response lacking a critical condition, not an error. Therefore, a different approach is needed: a representative set of test cases, acceptance criteria instead of precise answers, and measurement across several distinct dimensions.

Separating these dimensions is what makes testing useful. "The answer isn't good" isn't a finding; "the topic was correctly identified, but the relevant source wasn't retrieved" is a fixable finding.

The Four Dimensions of Measurement

Building a Representative Test Set

The source material should be real customer interactions, not scenarios crafted in a meeting. Transcripts, emails, and Case descriptions capture what's missing from invented scenarios: typos, partial phrasing, two questions in one sentence, and missing information.

A recommended breakdown: about half common cases, about a quarter edge cases—exceptions, borderline eligibility conditions, multi-part questions—and about a quarter cases intentionally designed to fail: out-of-scope requests, attempts to extract unauthorized information, and customers demanding a human agent.

For each case, define four fields: the actual inquiry as phrased, the expected topic, the acceptance criterion for the response, and the expected process behavior—including "should escalate" as a valid outcome, not a failure.

Acceptance Criteria Instead of Precise Answers

This principle is what makes testing possible at all. Instead of writing the "correct" answer, create three short lists: facts that must appear, statements that are forbidden from appearing, and the source that must be cited.

A typical example: a question about refund eligibility. The timeframe and product condition must appear; a commitment to a refund must not appear; the source must be the current version of the return policy. Two completely different phrasings could both pass.

This is also the format that enables reliable automated scoring—checking for the presence of a precisely defined fact is far more accurate than asking a model to evaluate "quality."

Automation vs. Human Judgment

Automated scorers are suitable for topic identification, presence of a valid citation, format compliance, length, and identifying forbidden statements. These run on the entire set for every version, at a low cost.

Human judgment is required for content accuracy in sensitive areas and for customer-facing phrasing. The entire set isn't needed—a fixed sample of twenty to thirty cases per version, selected to include edge cases, is sufficient.

The danger of relying solely on an automated scorer is a bias towards responses that sound authoritative. A persuasive answer that omits an eligibility condition will pass automatically but fail with a human reviewer.

How test results link to production monitoring is explained in Observability for Agentforce.

Edge Cases to Always Include

A question with two topics in one sentence. An inquiry missing critical information—does the agent ask clarifying questions or guess? A customer phrasing in a negative tone—is escalation triggered? A request for an action not permitted for that user. A question about a non-existent product—does the agent admit it or hallucinate? Content containing a disguised instruction attempting to alter behavior.

These six cover most failures observed in production, and they are inexpensive to re-run.

Bypass scenarios tie into the security tests detailed in Agentforce Security and Shared Responsibility.

Go-Live Thresholds

The threshold isn't a single number but is derived from the channel and risk. An internal agent assisting a representative can go live with a lower accuracy level because the representative filters the output. A customer-facing agent requires a significantly higher threshold, and crucially, zero failures in critical categories.

The rule more important than the number: zero failures in mandatory categories. Exposure of unauthorized information, irreversible action without approval, failure to escalate a direct request for a human—each of these blocks go-live regardless of the overall score.

Scenario: A Small Set Prevented a Bad Launch

A tourism company planned to launch a customer agent after an internal pilot showed good performance. The test set included 90 cases, 22 of which were edge cases from real inquiries.

The run revealed a pattern: for questions about changing dates with special cancellation conditions, the agent provided a mostly correct answer but omitted change fees in a third of the cases. This wasn't caught in internal tests—representatives knew to add the information themselves.

The launch was delayed by three weeks. The fix was in the content: billing conditions were moved to a separate, clearly marked section in all relevant articles, and an explicit acceptance criterion was added. The re-run passed, and the launch proceeded without incident.

Risks and Preventative Actions

Test Metrics

For assistance in building a test set and defining go-live thresholds, Agentforce and AI Services is the practical next step.

Testing Checklist

• ☐ Test set built from real inquiries
• ☐ Composition includes common, edge, and intentionally failing cases
• ☐ Each case has acceptance criteria: mandatory, forbidden, source
• ☐ Measurement is split into topic, retrieval, response, and process
• ☐ Automated scorers run on the entire set
• ☐ A fixed human sample is reviewed in each version
• ☐ Bypass scenarios and disguised instructions are included
• ☐ Mandatory categories with zero tolerance are defined
• ☐ The set runs as regression before every version release
• ☐ Test results are documented and compared to the previous version