Fail Open or Fail Closed for LLM Security Controls

Every security check you add is a new dependency. It can be slow, it can return an error, and it can go down during your busiest hour. The question is what your application does at that moment, and the answer should be a deliberate decision, not whatever the HTTP client does by default.

The two extremes

• Fail open means that if the check cannot produce a result, the request proceeds. Users keep working, but the request is unchecked.
• Fail closed means that without a result, the request is blocked or held. Nothing unchecked gets through, but the feature may stop.

Neither is correct in general, and the answer applies to any LLM firewall or scanner you add. A blanket fail-open policy turns any outage or timeout into a way around your controls, and an attacker who can make the scanner slow can bypass it. A blanket fail-closed policy can turn a scanner hiccup into a full outage of a customer-facing product.

Decide by consequence

Classify each workflow by what happens if an unchecked request goes wrong, and by whether the result can be undone.

Workflow	Impact if unchecked	Suggested behavior when the check fails
Public FAQ chatbot, no tools, no private data	Low: worst case is an embarrassing answer	Fail open with an alert, or fall back to a cached or canned response
Internal copilot reading company documents	Medium: possible data exposure	Fail closed for the retrieval-backed answers; degrade to a no-retrieval mode
Agent that sends email, spends money or deletes data	High and hard to reverse	Fail closed, or route to human approval
Batch classification with no user output	Low, and retriable	Queue and retry; do not release results until checked

The dimensions are the ones that matter in the tiered approval model: impact and reversibility.

Design the failure path

1. Set a timeout that reflects the user experience. Choose it from measured latency, not from the default of your HTTP library. Include the tail, not the average. See balancing latency, false positives and risk.
2. Retry sparingly. One quick retry is fine. Aggressive retries during an outage create a retry storm that makes the outage worse. Use backoff and a circuit breaker.
3. Define a degraded mode. For medium-risk workflows, "fail closed" does not have to mean an error page. It can mean answering without retrieval, disabling tool use, or showing a message that the assistant is temporarily limited.
4. Route uncertain cases to review. A scanner that returns "uncertain" is different from one that is down. A review queue gives you a third option besides allow and block.
5. Never fail open silently. Emit an event every time a request bypasses a check, with the workflow, reason and request ID, and alert on the rate. Unlogged fail-open is indistinguishable from having no control.

Test it on purpose

Fallback code that has never run will not work when needed. In staging, and on a schedule:

• Make the scanner return errors and slow responses and check that each workflow does what its policy says.
• Confirm alerts fire and that the event includes enough detail for logging and investigation.
• Check that an outage in the scanner does not take down unrelated workflows.

Common mistakes

• One global fail-open switch for every product.
• Retry storms that amplify an outage.
• Failing open with no telemetry or alert.

Sources and further reading

• NIST AI 600-1: Generative AI Profile