Cut critical AI agent errors by 90% and kept the product running through an LLM provider outage with error monitoring, smart retries, multi-vendor fallback, and alerting.
The AI agents had no resilience built in: no retries, no connection reuse, and no fallback. Every request depended on a single LLM vendor, so any provider hiccup reached users directly as a failure. Errors weren't categorized either, so the team couldn't tell which failures mattered most.
Approach
Measure first. I set up full error monitoring in LangSmith, grouped failures by error type, and ranked them by severity and product impact. That ranking decided what to fix first.
Shorter feedback loop. Errors and bugs now surface with enough context that issues move from customer report to product and engineering much faster.
Retries with jitter, on retryable errors only. Rate limits and transient failures are retried with backoff and jitter. Non-retryable errors fail fast instead of wasting tokens and latency.
Connection reuse across requests instead of a new connection per call.
Multi-vendor fallback through LiteLLM, which removed the single point of failure on one LLM provider.
Slack alerting so the team hears about error spikes before customers do.
Request path: retries and failover in the LiteLLM router, with errors traced in LangSmith and alerted in Slack
Impact
90% reduction in the critical error rate.
During a major outage at one LLM provider, traffic failed over automatically with no noticeable impact on users, so the outage never became ours.
Error spikes now reach the team in Slack before customers report them.
