Fail Secure vs Fail Safe in MCP Gateway Design

Anthropic released the Model Context Protocol in late 2024, and the security community has been playing catch-up ever since. MCP connects AI agents to enterprise tools and data through a client-server architecture. What the protocol does not do, by design, is enforce security at the protocol level. That gap is exactly where the gateway lives, and it is where the interesting problems start.

A gateway is a centralized middleware layer. Every agent request passes through it before reaching an MCP server. Inside that layer, the gateway applies identity-aware access controls, validates tool schemas at discovery and again at runtime, audits every tool invocation, and filters what the agent can see and call. Think of it as the last adult in the room before an agent touches something consequential.

The failure modes worth naming are not exotic edge cases. The policy engine becomes unreachable due to a network partition or crash. The identity service times out mid-request. A schema validation step encounters a malformed tool definition. A new tool gets dynamically injected mid-session by a server the gateway already approved. A circuit breaker trips under load and the gateway cannot evaluate the call in time. Each of these is a distinct moment that demands a pre-configured answer, and most teams have not pre-configured one. That absence is itself a configuration.

Here is where the terminology matters and where people get it wrong. "Fail-safe" sounds like the more secure option. It is not. Fail-safe, or fail-open, protects people: no one gets trapped, no process gets hard-blocked, access defaults to open. Fail-secure, or fail-closed, protects assets: data and systems stay inaccessible when trust cannot be established.

The classical physical analogy makes this concrete. A fire-door lock that releases on power failure is fail-safe; the door opens so people can escape. A vault lock that seals on power failure is fail-secure; the contents stay locked so assets are protected. Neither design is universally better. They answer different questions about what the system is built to protect when it breaks. NIST SP 800-160 Vol. 1 defines fail-secure as "a mode of termination of system functions that prevents loss of secure state when a failure occurs." The operative phrase is "prevents loss of secure state," not "prevents all damage." That distinction will come back around.

The gateway is a policy enforcement point. Its failure mode is itself a policy decision, whether the team made it deliberately or not.

The Threat Environment That Makes the Default Choice Consequential

The baseline exposure here, before any gateway failure is even considered, is not theoretical. Censys found 12,520 publicly exposed MCP services, roughly 40% with no authentication at all. Among servers that do enforce authentication, 53% rely on static API keys, single long-lived credentials with no expiration and no per-operation scope. VIPER-MCP scanning of real-world MCP repositories found 106 zero-day vulnerabilities. Adversa AI scanning found 38% of servers with no authentication on critical endpoints and 43% vulnerable to command execution. GitGuardian documented 24,008 unique secrets exposed in MCP configuration files in 2025 alone, Google API keys and database connection strings among the most common categories.

Named incidents make the abstraction land harder. CVE-2025-49596, carrying a CVSS score of 9.4, enabled arbitrary command execution through unauthenticated MCP Inspector instances. The first confirmed malicious MCP package appeared in September 2025, ran undetected for two weeks, and exfiltrated email data throughout. Over tens of thousands of exposed OpenClaw instances were identified in a four-day window in early 2026, hundreds of them running unauthenticated MCP endpoints leaking API keys and Slack credentials.

Against that backdrop, a gateway that fails open hands the attacker, or the misbehaving agent, exactly the access the gateway existed to prevent. That is not a worst-case scenario. That is the predictable outcome.

There is also a compounding factor that gets underappreciated in most architecture discussions. A single agent task can trigger calls to many different MCP tools, and those calls are not always visible upfront; they are generated as the agent works through the task. The blast radius of a fail-open moment is therefore not bounded by what the user originally requested. It is bounded by whatever the agent can reach, which in a poorly scoped deployment is an open question. That is the environment in which your default failure posture gets exercised.

Why Fail-Secure Is the Right Default for Most MCP Gateway Configurations

If trust cannot be established, deny the action. The threat environment above is why this applies broadly, not only to deployments someone has already labeled high-risk.

Default-deny means explicit allowlisting: tool names and versions, parameter schemas with permitted types, approved scopes. Anything not on the list is blocked before evaluation, not evaluated and then failed. That distinction matters more than it sounds. The goal is to eliminate the evaluation surface for unauthorized requests entirely, not to catch them after they have already touched the policy engine.

The two-checkpoint architecture makes fail-secure at the second gate non-optional. Validating tool schemas at discovery is necessary but not sufficient. A model can still be manipulated into calling an approved tool with malicious arguments, a class of attack the OWASP MCP Top 10 formalizes under context injection and confused deputy categories. The gateway must re-validate at runtime against the same schema. If the model attempts to pass an unexpectedly large payload to a query field, that gets caught at the second gate, not after the database query executes.

Per-call authorization also matters for a reason that gets overlooked. Long-running agents authenticate at session start and then keep operating. Permissions can be revoked after a session begins. A gateway relying on session-level trust will honor a permission that no longer exists. Per-call authorization means every tool invocation is a fresh authorization decision, which is the only posture consistent with zero trust architecture.

Here is the part that often gets resistance in engineering conversations: policy evaluation at the gateway is deterministic; model adherence to safety instructions under adversarial conditions is not. Research on large language model behavior under red-team conditions, including work published by Anthropic and academic groups studying prompt injection, consistently shows that probabilistic systems do not provide the same reliability guarantees as rule-based enforcement. Building governance on a probabilistic layer when a deterministic one is available introduces unnecessary risk. That is not a philosophical position. It is an engineering constraint. The model is willing, but the policy is weak.

Regulatory and framework alignment reinforces the same direction. NIST SP 800-207's Zero Trust architecture, joint guidance published by CISA, NSA, and international partners, and the OWASP MCP Top 10 all converge on minimum necessary access, verification at every step, and designing for containment before efficiency. Fail-secure at the gateway is the mechanism through which most of those requirements get enforced simultaneously, not a point solution for any one of them.

One specific case worth naming: if a server attempts to introduce a new, unapproved tool during an active session, the correct response is to treat it as a fresh discovery event, block the tool pending validation, and pause the agent. That is not paranoia. That is how dynamic injection attacks get stopped.

Where Fail-Open Behavior Has a Legitimate Place in the Design

The availability tension is genuine. A fail-secure control that blocks without a recovery path will push teams toward shadow workarounds, and when the governed path becomes unreachable, users route around it. Governance collapses in practice regardless of what the policy documents say. I have watched this happen, and it is not a theoretical failure mode.

DDoS conditions illustrate the real design tension. When a gateway cannot distinguish real requests from attack traffic, blocking all of it punishes legitimate users. Allowing authenticated sessions through while blocking unauthenticated traffic is a documented pattern for this specific scenario, a scoped application of fail-open logic to a defined condition.

Low-stakes, high-availability tools warrant a different calculus too. A tool that retrieves public documentation carries a different blast radius than one touching financial records or write-capable APIs. When a gateway blip blocks a documentation retrieval tool, the consequence is a degraded user experience. When it blocks a tool with write access to a regulated data store, failing open is a potential compliance incident. Treating those as the same situation is where design debt accumulates.

Fail-open is appropriate when the cost of denial exceeds the cost of worst-case access. Fail-secure is required when the reverse is true. The gateway designer's job is to assess this per tool, per scope, not once for the whole system. Different tool categories within the same gateway can carry different failure behaviors, and those behaviors need to be documented before production, not discovered during an incident when someone is already on the phone with legal.

The Operational Design Decisions That Make Fail-Secure Survivable

The single-point-of-failure objection to fail-secure is valid, and it is solvable. A gateway that fails secure but has no high-availability design will eventually cause total agent blackout. The answer is to make gateway failure rare through horizontal scaling, per-tenant quotas, and circuit breakers between agents and the gateway. Reduce the frequency of failure events; do not soften what happens when they occur.

Latency overhead is real but context-dependent. Some published gateway implementations add latency in the low-millisecond range at p99 for policy evaluation; others add more, particularly on cold-start paths. For conversational agents where response time is visible to users, this matters and should be benchmarked before production. For background automation workflows, it typically does not. Measure it. Do not assume.

Microsoft's open-source gateway work offers a concrete implementation model worth studying closely. The approach to the bash tool is illustrative: a scrubbed, default-deny process environment; only a minimal allowlist of environment variables passed to the spawned shell; a regex denylist for dangerous shell operations. That is defense-in-depth expressed in code, and it demonstrates how fail-secure defaults get translated from policy language into enforcement logic at a granular level.

Recovery path design is part of the fail-secure decision, not an afterthought. What fires when the gateway blocks? Who gets paged? How long does an agent loop pause before an operator is notified? Is there a documented manual override path, and who has authorization to use it? These questions need answers before deployment. The teams that leave them unanswered will answer them anyway, under duress, during an incident.

Observability is the prerequisite that makes all of this functional. A fail-secure gateway that blocks silently is nearly as dangerous as one that fails open. Operators need real-time signals, not post-incident audit logs, to distinguish a legitimate block from a misconfigured policy. Observability is load-bearing infrastructure here, not a nice-to-have.

Making the Architectural Decision Deliberately Before a Failure Forces It

The central risk is not picking the wrong default. It is not picking one at all. An undocumented gateway behavior under failure is still a choice, made by whoever wrote the code, not by the security team.

For each connected MCP server, the questions are concrete: what is the blast radius of an unintended tool call? What is the blast radius of denying that call, in terms of agent pause, user-facing error, or cascading workflow failure? Is the tool write-capable or does it touch regulated data? If yes, fail-secure by default. Is it read-only against non-sensitive data? It tolerates a more permissive failure posture. What is the recovery SLA, and what happens to in-flight agent tasks in the interim?

A registry is a prerequisite for this analysis to be executable. Every MCP server in use needs a registry entry with its failure posture documented before the gateway goes to production. Discovering that posture during an incident is not discovery. It is exposure.

Governance is what makes the decision stick over time. A fail-secure default set at deployment can drift as new MCP servers are added without going through the same assessment. Policy enforcement needs to be continuous and auditable, not a one-time configuration exercise the team revisits only when something breaks.

Teams that make this decision before deployment, and build the high-availability, observability, and registry infrastructure around it, are the ones that can authorize production deployments with confidence. The friction people attribute to security controls is almost always the absence of a gateway policy, not the presence of one.