Third-Party API Outage That Was Mistaken for Own Code
Four hours of debugging internal code before realizing the root cause was a third-party geolocation API silently returning stale data.
Overview
A geolocation vendor quietly started answering "US" for every ZIP code, and international orders were priced as domestic while every test still passed.
What happened
Support tickets spiked: shipping costs for international orders came back as zero. The team spent the first two hours looking at their own code. The answer turned up in hour three.
- Recent deployments — Last deploy was 72 hours earlier
- Pricing service code — Reviewed, nothing wrong
- Replayed requests in staging — Passed
- Integration tests — Passed
Why it broke
The vendor changed its fallback. Instead of returning an error for a ZIP code it could not resolve, it returned "US". The response was still HTTP 200 with a country code, so the contract had not formally changed and the integration test passed.
> [!WARNING] No breaking-change notice was sent. A same-status, same-schema response can still mean something different.
The fix
Detect the meaning of the response, not just its shape.
Key concepts
- Silent behavioral change
- A dependency change that does not alter the HTTP status code or schema but changes the semantic meaning of the response, breaking consumers without causing obvious errors.
- Consumer-driven contract testing
- Tests where the consumer (caller) defines what it expects from a provider, and the provider runs those expectations as part of its own test suite (e.g., Pact framework).
- Canary value monitoring
- Sending known test inputs to a dependency and alerting if the output deviates from expected — useful for detecting silent behavioral changes in third-party APIs.
When to use it
- Discussing API contract testing and consumer-driven contracts
- Explaining observability requirements for third-party integrations
- Justifying defensive integration patterns (validation, alerting on unexpected values)
Interview questions
- How would you design monitoring to detect silent behavioral changes in a third-party API?
Sign up free to read the full topic
More War Stories topics
- NAT Gateway Port Exhaustion Under Bursty Lambda Traffic
- Node.js Memory Leak That Required Daily Restarts
- Public S3 Bucket Exposed Customer PII for 8 Months
- Silent Data Corruption from an Off-By-One Migration
- Single-Region Dependency That Caused a Global Outage
- The Cascade That Took Down Three Services
- Thundering Herd After a Cache Cluster Restart
- TLS Certificate Expiry That Took Down an Internal API
Start free on Cariara · Coding problems · System design studio