Degradation and resilience

In real environments failures are normal: provider rate limits, embedding timeouts, an index that is not ready, deleted material, exhausted budgets, user cancellation. Resilience does not mean every request succeeds. It means failures stay explainable, recoverable and free of fabricated results.

Classify the failure first

Failure	Typical cause	Retryable
Model call	Timeout, rate limit, 5xx, authentication failure	Rate limits and transient faults may retry with a bound; authentication does not
Tool execution	Invalid input, unauthorized, business rejection, side-effect failure	Read-only queries may retry; writes need an idempotency key
Retrieval	Embedding timeout, index not ready, no hits	May fall back to lexical or a whole-document read
Corpus	Missing file, empty corpus, stale revision	Do not retry; state that the material is unavailable
Budget and cancellation	Context overflow, user cancellation, deadline reached	Do not retry; end under the original contract

The class decides the action. Treating every failure as "try again" amplifies rate limits, duplicates side effects and hides real configuration problems.

Three responses

1. Bounded retry: only for transient, idempotent, cancellable calls. Bound attempts, interval and total deadline, and keep propagating signal.
2. Degrade: narrow the capability while preserving real sources and identity. For example, fall back to lexical search when retrieval fails, or to a whole-document read when vectors are unavailable.
3. Fail explicitly: when no safe fallback exists, state the failure or ask a clarifying question instead of inventing a result.

Degradation never means switching to a looser permission path or bypassing validation. Identity, authorization, budgets and cancellation stay in force on every fallback path.

A handling matrix

Scenario	Recommended action	Avoid
Model rate limit	Bounded retry with the original request and cancellation	Unbounded retries or an unauthorized model
Model authentication failure	Fail with a clear error and record the configuration source	Silently falling back to a generic answer
Invalid tool input	Return a validation error and clarify if needed	Rewriting the user's intent and executing again
Write timeout	Reconcile the real commit using the idempotency key	Retrying blindly and writing twice
Embedding failure	Fall back to lexical search	Presenting stale vectors as new results
Index not ready	Whole-document read or lexical search	Treating "not ready" as "no material"
No retrieval hits	State that no material exists	Filling the gap with general knowledge
Budget exceeded	Fewer candidates or reads in passes	Truncating rules or evidence
User cancellation	Stop further calls and keep the reason	Recording cancellation as success

Timeouts and cancellation

• Give each layer its own timeout and make the total deadline cover the whole chain. An inner timeout should not outlive an outer one.
• Propagate cancellation through signal. External side effects that already happened cannot be pretended away.
• Timeout and cancellation are different outcomes. Timeouts may retry; cancellations usually should not.
• When a stream is interrupted, do not send a fabricated successful terminator.

What not to do

• Loosen authorization, skip source checks or trust client-asserted identity to raise the success rate.
• Turn retrieval failure into "there is no such material", making users believe the fact does not exist.
• Treat stale index copies, old caches or a previous answer as current facts.
• Keep calling tools or models after cancellation and discard late results.
• Cover every failure with one canned reply, making the problem invisible.

Observability

Record the failure stage, tool or retriever ID, error code, attempt count, whether degradation happened, its reason, total latency and cancellation reason. Never log secrets, full prompts or unmasked personal data.

Logs should answer "why did this request degrade", not only "did it succeed". When degradation rises, check dependency health before tuning prompts.

Verify with MockGPT

Failure paths should be reproducible. Offline verification with MockGPT can inject HTTP errors, network rejection, stream interruption and cancellation to confirm that:

• Retry counts are actually bounded;
• Degradation really happens instead of a silent success;
• A failed write is reconciled using its idempotency key;
• Cancellation stops subsequent calls.

Health checks for real dependencies belong to deployment. Simulation only verifies code branches.

A decision order

type Outcome =
  | { kind: 'answer'; sources: string[] }
  | { kind: 'unknown'; reason: 'no-material' }
  | { kind: 'degraded'; reason: 'retrieval-unavailable' }
  | { kind: 'failed'; reason: string };

async function answer(question: string): Promise<Outcome> {
  const sources = await retrieve(question).catch(() => undefined);
  if (!sources) return { kind: 'degraded', reason: 'retrieval-unavailable' };
  if (!sources.length) return { kind: 'unknown', reason: 'no-material' };
  return { kind: 'answer', sources };
}

The example only shows the order: separate "retrieval unavailable" from "no material", then choose answer, degradation or failure. Real authorization, budgets and cancellation checks still happen through their own interfaces.

Common issues

Symptom	Cause	Fix
More requests during rate limiting	Retries are unbounded or lack backoff	Bound attempts, interval and total deadline
A failure looks like "no material"	Degradation reasons are not separated	Distinguish retrieval-unavailable from no-material
Writes happen twice	A timeout was retried directly	Use an idempotency key and reconcile the commit
Output continues after cancellation	signal is not propagated	Check cancellation before every layer call
Degradation goes unnoticed	Only success metrics are recorded	Track degradation and failure reasons separately

Next

• Inject failures for regression: Offline verification with MockGPT.
• Define the corpus policy when retrieval fails: Building a RAG corpus.
• Confirm failure paths before release: A soft checklist before production.