Designing an API Gateway

An API gateway is a system people dismiss as "a reverse proxy with auth on it." That description is to a gateway what "a box that stores data" is to a database. This post designs one the way an interview expects: start with clients talking directly to services, watch that architecture break, and add only the layer each break demands. By the end you'll see why every piece is there — and why a gateway is simultaneously the most powerful and most dangerous component in the system.

Step 0 — Clarify the problem before you design

Before drawing anything, agree on what a "gateway" means here, because the word covers three different products. Clarify it down to the case the interviewer means: an inbound gateway for external clients (web/mobile/third-party) to reach many internal services. That is north-south traffic — distinct from service-to-service (east-west) mesh gateways, which solve a different problem.

Three things to state out loud, because they scope everything after:

1. The problem is cross-cutting concerns, not routing. Many services each need auth, rate limiting, logging, and TLS. We either copy that into every service, or we put it in one place. The gateway is that one place.
2. It is a chokepoint by design. Every request flows through it. That is the source of its power (one place to enforce policy) and its danger (one place that can take everything down).
3. Latency budget. The gateway adds a hop. Agree on a budget — a few milliseconds — because every feature below competes for it.

State the framing: "I'm designing a north-south gateway that centralizes cross-cutting policy for external clients. If we meant an east-west service mesh, the design is different." Half of system design is refusing the wrong problem.

Step 1 — The naive world: clients call services directly

Start with no gateway. Each external client knows the address of each service and calls it directly.

This fails in three independent ways, and each one motivates a different layer:

• No uniform policy. Every service re-implements auth, rate limiting, and TLS in its own way. They drift. One service forgets a check, and it's a breach. → motivates centralized policy.
• Clients coupled to service topology. The client knows IPs and ports that change every deploy. Add, split, or move a service, and every client breaks. → motivates a single stable entry point.
• No place to shape traffic. When service C is overloaded, there is nowhere to throttle or shed load before it dies — every client just hammers it. → motivates a control point for traffic.

All three point the same way: put one thing in front of everything.

Step 2 — The single entry point: one URL, many services

Introduce a gateway. Clients talk to one address; the gateway forwards to services. The immediate wins are real but shallow:

• Clients depend on one stable endpoint, not shifting service topology.
• There is now a place to add policy.

But routing is a feature, not the job. The job begins the moment we ask: now that every request passes through here, what must we do to every request, before it does any work? The answer is a pipeline, and the first stage is identity.

Step 3 — Authentication: the first mandatory policy

The gateway's first real responsibility is who is this? Before a request reaches a service, the gateway validates the caller.

• Verify credentials centrally — API keys, JWTs, mTLS. Services no longer each implement token parsing; they receive a request already authenticated, with identity attached in a trusted header.
• Reject early. An unauthenticated request that is dropped at the gateway never touches a service. This is the gateway protecting the system behind it, and it's why "fail early" is the gateway's guiding principle.

If your services are each still parsing tokens, you have a reverse proxy with ambitions, not a gateway. The policy only counts if it is the policy the system actually relies on — which means bypassing the gateway must be impossible.

Authentication answers "who." The very next question — "may they do this?" — exposes the first design fork.

Step 4 — Authorization, and where it should live

Authorization is subtler than authentication because part of it belongs at the gateway and part does not. Decide deliberately.

• Coarse authorization at the gateway — "is this principal allowed to reach the payments service at all?" This is a property of the route and belongs here.
• Fine-grained authorization at the service — "may this user transfer this amount from this account?" This is a property of business logic and belongs with the data, not in a network component.

Push fine-grained rules to the gateway and you create two problems: the gateway gains a dependency on your data, and your security policy now has a network hop in front of it. Keep the gateway's authorization coarse and fast.

Now every authenticated, coarsely-authorized request is about to reach a service. The next failure mode is not about identity — it's about volume.

Step 5 — Rate limiting: the only thing between you and a bad client

This is the feature teams under-build and the one that saves you in production. The naive version — limit per IP — is wrong, and understanding why is the whole interview.

• Why not per-IP. IP limits punish thousands of users behind carrier-grade NAT, while a single abusive tenant on one IP gets a whole IP's worth of quota. The correct key is the authenticated principal, not the source address.
• Window choice is a fairness choice. A fixed window is cheap but bursty at the boundary (a client gets 2× quota by straddling the reset). A sliding window is smooth but costlier. Most abuse is stopped by either; most business disputes are caused by the boundary burst.
• Distributed state is the real cost. A local counter per gateway instance is trivial — and wrong the moment you load-balance across instances (each allows the full limit, so the real limit is N×). A correct limit needs shared state, which makes a rate-limit store (typically Redis) one of the most queried datastores in your system. Plan for it.

The principle: reject before work. A 429 returned at the gateway costs microseconds; a request that reaches an overloaded service costs a timeout and a victim. The entire point of the limit is to fail cheaply.

We now have a pipeline — auth, authorize, limit, route — that protects the services. The final, cruel twist is that the component built to protect everything is itself the most dangerous single point of failure.

Step 6 — Don't become the bottleneck

The gateway is a chokepoint by design (Step 0). So its own failure modes can take down the very system it protects. Three properties keep it honest:

• Stateless. Every gateway instance must serve any request. In-memory session state or rate-limit counters turn an instance into a pet; if it dies, those limits reset or those sessions drop. Push shared state out to a fast store and keep the gateway process disposable.
• Nothing synchronous off the critical path. Logging, metrics, audit writes — these must be buffered and flushed asynchronously. A gateway that blocks a response on a slow metrics endpoint becomes the latency it was meant to remove.
• Backpressure is mandatory. A gateway that doesn't propagate upstream saturation will queue requests until it exhausts memory — taking down the front door along with the back. Circuit-break or shed load before you queue.

A gateway that fails open lets bad traffic through when it breaks; one that fails closed takes the whole site down. Decide which failure you prefer, per policy, in advance — not in the incident channel.

Step 7 — Versioning and transformation: absorb change at the edge

There is one more job the gateway is uniquely positioned to do, and it's the ugliest part of running services: change the clients haven't caught up to.

• Translating an old request shape into a new one, or a new response into an old one, lets backend services evolve freely while clients move at their own pace.
• This is real leverage — and it's technical debt with a forwarding address. Treat shims as things you will eventually delete, and instrument how many requests still use each version, or they become permanent.

Step 8 — Measure what only the gateway can see

Because every request passes through it, the gateway is the one place with a complete view of traffic. A well-instrumented gateway tells you things no single service can.

• Cross-service latency — which routes are slow across the system, not just in one service.
• Client behavior — which callers are abusive, which are failing, by principal (not IP).
• p95 / p99, not mean — the tail is where angry users come from, and the gateway is where you can actually see it.

Treat its metrics as the primary map of system health — and remember the limit its presence imposes: a problem upstream is only ever as visible as the gateway's instrumentation makes it.

When you don't need one

A gateway adds a hop, a failure surface, and a team's worth of policy to maintain. If you have one client and one service, you don't have a gateway problem — you have a config file. Reach for this pattern when the cross-cutting concerns you'd otherwise copy into every service become the actual cost, and the centralization genuinely earns its single point of failure. Even then, build it in the order above: entry point, then identity, then volume, then resilience — because the order is the argument.