When you have one server, every request goes to it. When you have ten servers, something has to decide which server handles each request — that "something" is the load balancer.
A load balancer is the traffic cop of a distributed system. It sits between clients and a pool of backend servers, and its job is to spread load evenly, hide failures, and keep things fast.
One-liner: A load balancer is a reverse proxy that distributes incoming requests across multiple backend servers according to a chosen strategy.
1. Why do we need a load balancer?
A single server hits hard limits — CPU, RAM, network bandwidth, single point of failure. A load balancer lets us:
| Problem solved | How the load balancer helps |
|---|---|
| Single server bottleneck | Spreads requests across many servers, multiplying throughput |
| Single point of failure | If one server dies, traffic is routed to the healthy ones |
| Uneven load | Smart algorithms (least-connections, weighted) keep servers balanced |
| Rolling deployments | New versions can be rolled out server-by-server with controlled traffic |
| Geographic scaling | Global LBs route users to the nearest data center |
If you only run one server, you don't need a load balancer. The day you run two, you do. Most real systems run tens to thousands behind one.
2. Where can a load balancer live?
| Location | Role | Examples |
|---|---|---|
| Client-side | The browser or app picks a server itself (often via DNS returning multiple IPs) | DNS round-robin |
| Edge / Network | A dedicated appliance or cloud service sits in front of all servers | AWS ALB, Nginx, HAProxy, F5, Cloudflare |
| Server-side | Each server runs a sidecar that participates in a coordinated pool | Envoy sidecars in a service mesh (Istio, Linkerd) |
In practice, most setups combine all three — DNS returns a regional LB IP, that LB hands off to an in-cluster LB, and the in-cluster LB talks to sidecar proxies on each pod.
3. L4 vs L7 load balancing
This is the most important distinction and a very common interview question.
| Aspect | L4 (Transport Layer) | L7 (Application Layer) |
|---|---|---|
| OSI layer | Layer 4 — TCP/UDP | Layer 7 — HTTP/HTTPS/gRPC |
| What it inspects | IP addresses, ports, protocol | URL, headers, cookies, body, gRPC method |
| Routing decision based on | Connection-level info | Request content |
| Throughput | Higher — does not parse payload | Lower — must parse and possibly buffer the request |
| Smart routing | Limited (mostly IP/port based) | Rich (path-based, host-based, header-based) |
| Typical use | TLS termination offloading, raw TCP services, very high throughput | Microservices routing, canary releases, URL/host-based routing, A/B testing |
Easy way to remember:
- L4 LB = "I'm routing packets, I don't care what's inside."
- L7 LB = "I'm routing requests, I read the URL and headers."
L7 gives you far more power but costs CPU and adds latency. L4 is dumb and fast.
Real-world example: an L7 LB can send /api/ to your backend API servers but /images/* to your static asset servers — based entirely on the URL path. An L4 LB can't do that — to it, both look like the same TCP connection.
4. Load balancing algorithms
The algorithm decides which backend gets the next request.
4.1 Static algorithms (don't look at server state)
| Algorithm | How it picks | Pros | Cons |
|---|---|---|---|
| Round Robin | Server 1, then 2, then 3, then back to 1 | Simple, fair | Ignores server load — slow servers get the same number of requests as fast ones |
| Weighted Round Robin | Same as round robin but stronger servers get more turns | Handles heterogeneous servers | Weights must be tuned manually |
| Random | Pick one at random | Surprisingly effective at scale | Can be uneven for small pools |
| Hash-based (IP / URL) | Hash a key (e.g. client IP) → always routes to the same server | Sticky by default, good for caching | Uneven if hash isn't uniform |
4.2 Dynamic algorithms (look at server state)
| Algorithm | How it picks | Pros | Cons |
|---|---|---|---|
| Least Connections | Pick the server with the fewest active connections | Great when requests have variable duration | Needs accurate connection tracking |
| Least Response Time | Pick the server with the lowest recent response time | Adapts to real performance | Can be noisy; needs tuning window |
| Weighted Least Connections | Like least connections, but stronger servers are preferred | Best of both worlds for heterogeneous clusters | More complex; weights must be set |
4.3 Consistent hashing (special case)
A smarter variant of hash-based routing. Used heavily in caches, CDNs, and sharded databases.
The problem it solves:
Imagine 3 cache servers using simple hash(key) % 3. You add a 4th server — now it's hash(key) % 4 — and almost every key maps to a different server. Every cached item is invalidated. Cache stampede. Disaster.
Consistent hashing arranges servers and keys on a "ring", so adding/removing a server only remaps a small fraction of keys.
5. Health checks
A load balancer that routes to a dead server is worse than no load balancer. This is why every LB runs health checks.
| Type | What it does | Example |
|---|---|---|
| Active (probe) | LB periodically sends a request to each server and checks the response | GET /health expecting 200 OK within 200ms |
| Passive (in-band) | LB watches real traffic — if a server starts failing on real requests, it gets demoted | 3 consecutive 5xx responses → mark unhealthy |
| Liveness vs Readiness | Liveness = "is the process alive?" (restart if not). Readiness = "can it serve traffic right now?" (route traffic if yes) | Common in Kubernetes with separate probes |
Important: health checks and the load-balancing algorithm are two halves of one job. The algorithm spreads load; health checks keep traffic away from servers that can't serve it.
6. Sticky sessions (session affinity)
Some apps store session state in the server's memory (not in a shared store). If a user's next request lands on a different server, their session is lost.
Sticky sessions = the LB routes all requests from the same client to the same server (usually via cookie or IP hash).
| Aspect | Sticky sessions | Stateless (no stickiness) |
|---|---|---|
| Pros | Works with in-memory sessions, simple | Servers truly interchangeable; easy to scale |
| Cons | Uneven load (one user with heavy state hogs one server); loses server affinity during deploys | Requires external session store (Redis, DB) |
Interview wisdom: if your design says "we use sticky sessions because that's easier" — push back. Modern systems overwhelmingly use stateless servers + external session store (Redis). Stickiness should be the exception, not the default.
7. TLS termination at the load balancer
HTTPS requests are expensive to decrypt. Most production setups terminate TLS at the load balancer — the LB holds the certificate and decrypts the traffic, then passes plain HTTP to the backend servers.
Client ──HTTPS──> Load Balancer (decrypts here) ──HTTP──> Server
▲
holds the certWhy do this?
- Backends don't need certs → simpler ops
- LB can centrally enforce security policies
- CPUs on backends are freed up for actual work
- L7 features (header inspection, routing by URL) work — they couldn't see encrypted content otherwise
The LB then usually re-encrypts traffic on the way to backends inside a private network (called TLS re-encryption or end-to-end TLS).
8. Global vs Local load balancing
| Type | Scope | How it routes | Example |
|---|---|---|---|
| GSLB (Global Server Load Balancing) | Routes users to the right region/data center | DNS-based geo, Anycast, BGP routing | AWS Route 53 latency-based routing, Cloudflare |
| Local LB | Routes traffic within a single data center | L4/L7 algorithms, health checks | AWS ALB, Nginx, HAProxy |
A real-world request often passes through both:
Tokyo user → GSLB (DNS) → Tokyo LB → healthy server in Tokyo data center
Berlin user → GSLB (DNS) → Frankfurt LB → healthy server in EU data center9. Load balancing in microservices
In a microservice system, a single user request often fans out to many services. A traditional perimeter LB isn't enough — you also need east-west load balancing between services.
| Approach | How it works |
|---|---|
| Client-side LB (smart client) | The calling service has a library (e.g. Eureka, gRPC client) that knows the list of healthy instances and picks one |
| Sidecar proxy | Each pod has a sidecar (Envoy) that handles load balancing, retries, circuit breaking — the app doesn't even know |
| Central LB per service | A dedicated LB sits in front of each service's replicas |
Service meshes (Istio, Linkerd) are basically load balancing as infrastructure — every request between services goes through a sidecar proxy that can do retries, timeouts, circuit breaking, encryption, and observability, all configured centrally.
10. Common pitfalls and trade-offs
| Pitfall | What goes wrong | How to avoid it |
|---|---|---|
| LB becomes SPOF | One LB goes down → entire site down | Run LB in HA pair (active-active or active-passive); use floating IPs / anycast |
| Sticky sessions hiding state bugs | App loses session state the moment a server dies or is rebalanced | Push session state to an external store (Redis) |
| Aggressive health checks flapping | Server marked unhealthy → restored → unhealthy → ... causing chaos | Use sane thresholds (e.g. 3 failures in a row, then cooldown) |
| L7 CPU bottleneck | L7 LB becomes the new bottleneck under heavy load | Scale horizontally, terminate TLS upstream, use L4 for bulk traffic |
| Uneven distribution with simple hash | Naive hash % N remaps almost everything when N changes | Use consistent hashing |
| Connection draining on deploy | Rolling deploys kill in-flight requests | Use connection draining / preStop hooks / canary releases |
11. Interview Questions
1. What is a load balancer and why do we need one?
A load balancer sits between clients and a pool of backend servers and distributes incoming requests across them.
It's needed because a single server hits CPU, RAM, and bandwidth limits and is also a single point of failure.
A load balancer gives you horizontal scalability, fault tolerance, and the ability to roll out new versions safely.
2. What's the difference between L4 and L7 load balancing?
L4 (Transport layer) load balancing works at the TCP/UDP level — it routes based on IP, port, and protocol, without inspecting the payload.
It's faster and cheaper but can't make routing decisions based on URLs or headers. L7 (Application layer) load balancing parses HTTP, can read URLs, headers, cookies, and body, and can route based on them (e.g. /api to one backend, /images to another).
It's more powerful but consumes more CPU and adds latency.
3. Compare round-robin, least-connections, and IP-hash load balancing.
Round-robin cycles through servers in order — simple and fair, but ignores server load and connection duration.
Least-connections sends new requests to the server with the fewest active connections — works well when requests have variable duration.
IP-hash hashes the client IP and always routes to the same server — gives sticky sessions for free but makes uneven distribution if client IPs aren't uniform.
4. What is consistent hashing and why is it useful?
Consistent hashing places both servers and request keys on a hash ring, and a request is routed to the next server clockwise.
When a server is added or removed, only about 1/N of keys need to be remapped, unlike naive hash % N where almost everything remaps.
This is essential for caches and sharded systems where remapping invalidates data and causes cache stampedes.
5. Why are health checks important for a load balancer?
A load balancer that routes traffic to a dead server is worse than no load balancer — users see errors instead of the system "degrading gracefully".
Health checks, such as active probes like /health and passive checks that monitor real traffic, let the load balancer detect failed servers and remove them from the pool until they recover.
6. What is TLS termination at the load balancer?
The load balancer holds the SSL/TLS certificate and decrypts incoming HTTPS traffic, then forwards plain HTTP to backend servers.
This centralizes certificate management, reduces CPU load on backend servers, and enables L7 features that need to inspect request contents, such as URL routing and header inspection.
Inside the data center, traffic is often re-encrypted for compliance and end-to-end security.
7. How do sticky sessions work, and what's the downside?
The load balancer identifies a user, usually by cookie or IP hash, and routes their requests to the same backend server.
This allows applications to store session state in server memory.
The downside is that load can become uneven, and if a server dies, that user's session may be lost. Most modern designs avoid this by keeping servers stateless and storing sessions in a shared store like Redis.
8. What's the difference between global load balancing and local load balancing?
Global Server Load Balancing (GSLB) routes users to the appropriate region or data center, typically using DNS-based geo-routing or Anycast.
Local load balancing then routes requests to a specific server within that data center using L4/L7 algorithms and health checks.
For example, a request from Tokyo might follow:
GSLB (DNS) → Tokyo LB → healthy Tokyo server9. Is the load balancer itself a single point of failure?
It can be — if you run only one load balancer.
In practice, load balancers are deployed as a high-availability pair, either active-active or active-passive, using mechanisms such as floating IPs, BGP, or Anycast.
Cloud providers expose regional load balancer endpoints, such as AWS ALB, and handle the underlying high availability internally.
10. How does load balancing work inside a microservices system?
A single public-facing load balancer at the edge is not enough — service-to-service communication also needs load balancing.
Common approaches include:
- A smart client library that maintains a list of healthy instances, such as Eureka or gRPC client-side load balancing.
- Sidecar proxies like Envoy, where every pod has its own proxy that handles load balancing, retries, and circuit breaking.
- A central load balancer for each service.
Service meshes such as Istio and Linkerd commonly use the sidecar-proxy approach.
11. How would you do a zero-downtime deployment using a load balancer?
Use connection draining and rolling deployments.
Add a new server running the new version to the load balancer's pool and send a small slice of traffic to it as a canary. Monitor errors and performance, then gradually shift more traffic to the new version.
When removing an old server, mark it as "draining" so the load balancer stops sending new requests while allowing existing connections to finish.
This prevents in-flight requests from being terminated during deployment.
12. What is a "cache stampede" and how does consistent hashing help prevent it?
A cache stampede occurs when many cached keys expire or are invalidated at the same time, causing a large number of cache misses and a sudden burst of requests to the database.
Consistent hashing reduces the impact of cache-server changes by ensuring that when a server is added or removed, only a small portion of keys are remapped.
This prevents the entire cache from being invalidated at once, reducing the risk of a thundering-herd effect against the database.
Summary
- A load balancer distributes traffic across multiple servers and hides failures.
- L4 routes on connection info (fast, dumb). L7 parses requests (powerful, costly).
- Algorithms range from simple round-robin to consistent hashing, each with different fairness and stability properties.
- Health checks keep traffic away from broken servers.
- Sticky sessions are easy but pushed-down on for being inherently anti-scale.
- Global + local LBs together route requests across the planet and within each data center.
- In microservices, sidecar proxies make load balancing a primitive every service gets for free.
Next up in a beginner's path: Caching — the second pillar of scale after load balancing.