When an MCP Gateway Earns Its Place — And Why It Has Nothing To Do With Server Count

The advice you will find repeated across a dozen vendor blogs is that a gateway becomes necessary somewhere past three MCP servers. It is a memorable number and it is measuring the wrong quantity. You can run ten MCP servers with no gateway at all and be entirely fine, and you can cross the threshold with one.

What actually forces the decision is statefulness, and specifically the moment you place more than one replica behind a single address. That is a deployment property, not a fleet size, and once you frame it that way the answer for most teams changes.

Why the MCP transport is stateful in the first place

The current draft specification is explicit that MCP is a stateless protocol: every request carries what is needed to process it, and a server must not infer context from earlier requests on the same connection or stream. The specification goes further and notes that an open connection — a STDIO process included — is not a conversation or a session. State spanning requests must be referenced by an explicit identifier the client passes on each call.

That removes the protocol-level session as a thing to route on, and it does not remove the operational problem. A server holding application state — an open working directory, an initialised subprocess, a partially completed job — still keeps that state somewhere. Under ordinary load balancing the follow-up request carrying the handle lands on a replica that holds no such state, and the handle resolves to nothing. What changed is the routing key: affinity on a protocol session identifier under 2025-11-25 and earlier, affinity on an application-level handle under the current draft, or shared state so affinity is unnecessary at all.

client                    k8s Service (round-robin)
  |                              |
  |-- tools/call (start job) -->  pod A   job H created, state in memory
  |<- handle H -----------------
  |
  |-- tools/call (handle H) --->  pod B   handle H resolves to nothing
  |                                       the state lives on pod A
  |
  |   the protocol no longer supplies a session to route on;
  |   H is an ordinary tool argument, so routing on it is
  |   application-level work the load balancer cannot do for you

This is the problem Microsoft's mcp-gateway exists to solve, and its documentation is precise about the mechanism: all requests carrying a given session identifier are routed consistently to the same server instance. That is session affinity, and without it a horizontally scaled stateful MCP server does not work — not intermittently, not under load, but structurally.

The four problems gateways are sold as solving

Vendor material tends to present a gateway as one product answering one need. It is more useful to separate the needs, because most teams have exactly one of them and buying a solution to all four means operating a component whose majority of surface area is unused.

Observability is the weakest of the four justifications

The last row is worth pausing on. Observability is the most common justification given for a gateway and the one it is least uniquely suited to provide. A gateway sees requests and responses; it does not see what a tool call cost in tokens, because the model call happens on the other side of the agent, not the other side of the gateway. If cost attribution is the goal, instrumenting the model client is both cheaper and more accurate.

Aggregation is a namespace problem in disguise

Putting several MCP servers behind one endpoint sounds like plumbing and is really a naming decision. Tool names are only unique within the server that declares them. Aggregate a GitHub server and a documentation server and there is nothing preventing both from exposing a tool called search.

The Envoy AI Gateway resolves this by prefixing tool names with the backend identifier, so clients see names like github__issue_read and context7__query-docs. It also merges the separate SSE streams from each backend into one and encodes several backend session identifiers into a single client-facing session, reconstructing event IDs so resumption still works across the merge.

Prefixed tool names become a public interface

That is a reasonable design and it has a consequence worth planning for: the tool identifiers your agent reasons about are now assigned by the gateway rather than by the server. Prompts that name tools explicitly, evaluation fixtures, and allow-lists all encode the prefix. Changing the backend's registered name later renames every tool it exposes, and anything that pinned those names breaks.

When you should not put a gateway in

The strongest case against is that a gateway is a second place for authorization to be wrong. It terminates the connection from the client and opens a new one to the server, which means the token boundary now has two sides and both need to be correct.

That interacts directly with the rule that an MCP server must never pass through the token it received. A proxy sitting in the middle is exactly the component most likely to relay a token because relaying is what proxies do. If your gateway forwards the client's bearer token to backend servers, you have reintroduced the confused deputy problem at the infrastructure layer — the failure discussed in why you cannot pass the client's token upstream, now applied to a component whose entire job is forwarding.

• One server, one replica, one client. Nothing here needs a gateway, and adding one is a component to run, patch, and page someone about.
• Stateless servers you have genuinely verified are stateless. If no in-memory state survives between calls, ordinary load balancing is fine and affinity buys nothing.
• STDIO transport. Sessions are process-scoped by construction; there is no network hop to route.
• Observability alone. Instrument the model client instead; it is the layer that can actually see cost.

Common mistakes

1. Scaling a stateful MCP server to two replicas without affinity and treating the resulting intermittent failures as a client bug. Check the replica count before you check anything else.
2. Assuming IP-based sticky sessions are sufficient. Multiple clients behind one NAT share an address, and one client may legitimately hold several sessions. Affinity must key on the session identifier the protocol provides.
3. Letting the gateway forward the client's bearer token to backends. Each hop needs its own credential, and a proxy is the component most likely to violate that by default.
4. Adopting a gateway for observability and discovering it cannot report token cost. It sits on the wrong side of the model call.
5. Treating backend registration names as internal. Once tools are prefixed, those names are in prompts and allow-lists and are effectively public API.
6. Adding a gateway before you have one stateful server behind two replicas. Solve the problem you have; the component is not free to operate.

Security and performance considerations

A gateway is a chokepoint, which is its security benefit and its blast radius in the same sentence. Every tool call in the fleet passes through it, so an allow-list applied there is genuinely comprehensive — and a compromise there reaches every backend. It should hold no long-lived backend credentials it does not need, and it should be the component with the tightest deployment controls, not the one nobody wants to touch because everything depends on it.

Affinity changes how you scale

Session affinity has a capacity consequence that surprises people. Pinning sessions to instances means load is distributed by session, not by request, and sessions are long-lived and unequal. One client running a heavy agent workload can saturate a single pod while its neighbours idle, and autoscaling on average CPU will not notice. Scale on session count per instance, or accept that headroom has to be provisioned per pod rather than per fleet.

Draining is the other operational cost. A rolling deployment that evicts a pod terminates every session pinned to it, and MCP sessions do not transparently migrate. Long grace periods help; designing servers so a lost session is cheap to re-establish helps more.

Troubleshooting

1. Intermittent unknown session errors — check the replica count first, then whether affinity keys on the session identifier rather than on source IP.
2. Notifications arrive sporadically or stop after a reconnect — the SSE stream reattached to a different instance, or Last-Event-ID is not being honoured through the proxy.
3. Tool not found after adding a second server — a name collision, or the prefix changed because the backend was re-registered under a different name.
4. Works in staging with one replica, fails in production — the classic signature of missing affinity. Staging did not have enough replicas to expose it.
5. Backend rejects the gateway's requests as unauthorised — check whether the gateway is forwarding the client token instead of presenting its own credential for that hop.
6. One pod hot while others idle — expected under session affinity. Scale on sessions per instance rather than on average CPU.

The decision is worth making explicitly rather than by default in either direction. Most teams reach for a gateway too early, on the strength of a fleet-size rule of thumb that does not describe their actual constraint, and a few reach for it far too late — after a week of chasing intermittent failures that a single line about replica counts would have explained. The cost of running one is real but bounded; the cost of scaling a stateful server without one is a failure mode that looks like everything except what it is.

If you are deciding how to deploy MCP servers and want the topology settled before it becomes an incident, that is the kind of architecture work our agent development team takes on. Where the surrounding platform work is the harder half — routing, identity, and the services around it — our API and integration engineering covers that side.