Why Integrations "Work" in Demo and Quietly Fail in Production
During standard user acceptance testing (UAT), you send a single message, confirm its reception, and approve. In production, that same integration processes thousands of messages daily, and some will inevitably fail—due to timeouts, row locks, expired authorizations, or schema changes in the connected system. The fundamental question for solution quality isn't "Does the integration work?", but rather, "What happens when it doesn't work, and who notices?"
Most of the costly failures I've observed didn't stem from bugs in the integration code itself, but from the absence of three critical capabilities: detecting a failed message, a mechanism to retry without creating duplicates, and a process to ensure data consistency between both systems by end-of-day. Without these, any integration "works" until the moment it's discovered it hasn't been working for two weeks.
Four Layers for Proper Error Handling
| Layer | What it solves | Typical failure without it |
|---|---|---|
| Idempotency | Re-running the same message doesn't create duplicate records | Duplicate orders or inventory movements after a retry |
| Retry with Backoff | Temporary failures (timeout, rate limit) are self-corrected | A momentary surge becomes a persistent issue |
| Dead Letter Queue | Non-temporary failures are flagged, not silently lost | A message is "swallowed," and parties assume it was processed |
| Business Reconciliation | Data discrepancies that didn't clearly fail are uncovered | A monthly report reveals a gap whose origin is hard to trace |
Each layer depends on the one before it. Retries without Idempotency create duplicates. A Dead Letter without Reconciliation masks that even "technically successful" messages didn't necessarily reflect the correct business state.
Idempotency: The Key to Preventing Duplicates
Any integration that can receive the same message more than once—and almost all integrations fall into this category—requires an External ID that uniquely identifies the event, not just the record. In Salesforce, the common implementation is an Upsert based on an External ID field with a unique constraint, combined with a logging table (Custom Object or Platform Event Log) that records which event identifiers have been fully processed.
Common mistake: relying solely on an Upsert to the business record itself (e.g., Order External ID) without documenting intermediate steps. If the process also involves updating inventory in an external system, an Upsert on the order doesn't prevent a duplicate call to update inventory. Every sub-operation with an external side effect needs to be idempotent itself, not just the final record.
Retry: Backoff Policy and Error Classification
Not every error warrants a retry. It's crucial to distinguish between three categories upfront:
- Temporary errors (Timeout, 503, Rate Limit) - Candidates for Exponential Backoff retries, meaning the interval between attempts increases (e.g., 30 sec, 2 min, 10 min) to avoid exacerbating system load.
- Structural errors (missing required field, validation rule violation, invalid value) - Should not be retried, as they will fail again in the same way. They should go directly to a Dead Letter Queue.
- Authorization or configuration errors (expired token, API Version change) - Require immediate alerts to a technical team, as they block the entire queue, not just a single message.
In Salesforce, retries are typically implemented at the middleware layer or within Apex Queueable/Batch jobs, with a retry counter stored on the record itself. A reasonable number of attempts for most cases is 3-5 with backoff, not infinite retries—unlimited retries turn a temporary glitch into sustained load on both systems.
Dead Letter Queue: Where Failed Messages "Live"
A Dead Letter Queue isn't just storage—it's a contract. Every message reaching it should include: the original event identifier, the full payload, a classified failure reason, the number of attempts made, and the time it entered the queue. Without this information, "handling" a Dead Letter becomes guesswork.
Two common implementation approaches in Salesforce:
- Dedicated Custom Object (
Integration_Failed_Message__c) with structured fields and List Views by error type—suitable when business teams need visibility within Salesforce itself. - External queue at the middleware layer (e.g., Dead Letter Exchange in MuleSoft/Boomi)—suitable when the technical team monitors outside Salesforce and wants to avoid burdening the Salesforce org.
The choice depends on who is expected to act on the failure: if it's a business process owner, they need to see it within Salesforce; if it's a technical integration team, an external layer is often preferable.
Business Reconciliation: The Check That Uncovers What Retries Missed
Even with perfect Idempotency and Retries, some failures "succeed" technically but create a business discrepancy—for example, a message received and processed, but with an incorrect value from outdated source data. Reconciliation is a periodic process (daily, hourly, depending on event volume) that compares counts or aggregate sums between two systems—for instance, the number of orders created in the ERP versus the number of orders created in Salesforce for the same day—highlighting discrepancies before they escalate into customer service issues.
A good Reconciliation process doesn't require field-by-field checking of every record; a checksum or aggregate count is sufficient to signal when deeper investigation is needed. In most organizations, daily frequency is adequate; for financial or critical processes (orders, invoices), checks within a few hours are necessary.
Decision Framework: When Each Layer is Mandatory, and When It's Optional
| Criterion | Idempotency required | Automatic Retry required | Separate Dead Letter required | Daily Reconciliation required |
|---|---|---|---|---|
| Event generates financial transactions or inventory movement | Yes | Yes | Yes | Yes |
| Event is one-way, read-only | Not critical | Yes | No | No |
| Volume over 500 messages per day | Yes | Yes | Yes | Recommended |
| External partner without high availability SLA | Yes | Yes, with long Backoff | Yes | Recommended |
| Integration between two non-financial, low-volume objects | Recommended | Recommended | Not essential | No |
The guiding principle for this table: the more a failure has financial or irreversible consequences (shipping, billing, inventory update), the more all four layers shift from "recommended" to "mandatory"—regardless of volume.
Example Scenario: Retailer with Two-Way Order Synchronization
A retail company with 40 branches uses Salesforce for B2B order management and an external ERP for inventory and invoicing. The integration was initially built with a simple REST call: when an order is created in Salesforce, a synchronous call creates it in the ERP. No retry, no Dead Letter.
During peak season (Black Friday), the ERP began returning timeouts for approximately 3% of calls. Without a retry mechanism, these 3% simply "vanished"—the order remained in Salesforce with a "sent" status, but the ERP had no knowledge of it. Within two days, about 140 orders accumulated that never reached the packing process, discovered only when customers called to inquire about their goods.
The solution implemented afterward: an Apex Queueable layer that retries up to 5 times with a backoff of 1/5/15/30/60 minutes; an ERP_Sync_Status__c field with values Pending/Synced/Failed; a Integration_Failed_Message__c Custom Object to consolidate final failures with a "Retry" button for the operations team; and a daily Reconciliation report that compares order counts between the systems and sends a Slack Alert when the discrepancy exceeds zero. The detection time for similar issues decreased from two days to less than an hour.
Common Risks and Preventive Actions
| Risk | How it manifests | Preventive action |
|---|---|---|
| Infinite retries on a structural error | The same message repeatedly fails, creating load | Classify errors upfront and send structural errors directly to Dead Letter |
| Lack of unique event identifier | Retry or duplicate call creates a duplicate record | External ID on the event, not just the final record |
| Dead Letter without an Owner | Messages pile up and no one addresses them | Assign an Owner and define SLA for resolution by event type, not system |
| Technical monitoring only (API status) | Integration is "green" but business data is inconsistent | Add Reconciliation comparing business outcomes, not just response codes |
| Fixed and too-short Backoff | Repeated attempts exacerbate load during widespread issues | Exponential Backoff with a defined maximum number of attempts |
Checklist Before Approving Error Handling Design
- ☐ Every event has a unique identifier (External ID) to prevent duplicates on retry.
- ☐ Errors are pre-classified as temporary/structural/authorization, with different handling for each type.
- ☐ Backoff policy is defined with a maximum number of attempts.
- ☐ An accessible Dead Letter Queue exists with full payload and failure reason.
- ☐ An Owner and defined SLA for resolution are established for each error type.
- ☐ A periodic Reconciliation process compares business outcomes between systems.
- ☐ Alerts are sent to a channel where someone actively reads them (not just logs).
- ☐ Test scenarios include service interruption of the secondary system, not just happy path.
How This Connects to the Rest of the Architecture
Error handling design doesn't stand alone—it relies on the data and permission layers defined in the CRM Architecture Guide, and on the decision of whether logic is implemented in Flow or Apex according to Salesforce Flow vs. Apex. The permission model through which integration components write data must be evaluated against the Salesforce Permission Model to ensure the technical integration user doesn't have overly broad access. And as message volume grows, the question of retries directly intersects with API limits detailed in Salesforce API Limits.
Summary
Integration error handling isn't a feature you add at the end—it's the difference between a system that breaks and is discovered after a customer complains, and one that flags issues before damage accrues. The four layers—Idempotency, classified Retries, a Dead Letter with an Owner, and Business Reconciliation—don't require a separate project, but they do demand explicit decisions during the planning phase, before the first integration goes live. An organization that self-alerts to 3% of messages failing within an hour is fundamentally different from one that discovers it from an angry customer.