Theory: LoadBalancer Deep Dive

From cloud controller reconciliation to MetalLB, annotations, traffic policy, and cost-aware Helm templating.

Introduction

Every Kubernetes cluster offers three Service types: ClusterIP (internal only), NodePort (static port on every node), and LoadBalancer. The LoadBalancer type is the most powerful -- it tells the cluster to provision a real, externally reachable load balancer. On AWS that means an ELB or NLB, on GCP a forwarding rule, on Azure a public or internal LB. Understanding how this provisioning works, how to configure it per cloud, and how to template it in Helm charts is essential for any production deployment.

This lesson covers the full lifecycle: how the cloud controller manager bridges Kubernetes and cloud APIs, which annotations you need per provider, how MetalLB solves the on-premises gap, and the trade-offs around traffic policy, session affinity, IP allocation, and cost.

How LoadBalancer Services Work

When you create a Service with type: LoadBalancer, Kubernetes does not provision infrastructure directly. Instead, it relies on the cloud controller manager (CCM), a component that runs in the control plane and implements the cloud-specific logic.

The reconciliation loop works as follows:

1. You apply a Service manifest with type: LoadBalancer.
2. The Kubernetes API server persists the object. The CCM watches for Service events.
3. The CCM detects the new LoadBalancer Service and calls the cloud provider API to create an external load balancer (e.g., CreateLoadBalancer on AWS).
4. The cloud provisions the LB, allocates an IP or hostname, and configures listeners and target groups.
5. The CCM writes the allocated address back into the Service's status.loadBalancer.ingress field.
6. External traffic flows: Client → Cloud LB → NodePort → kube-proxy → Pod.

If you delete the Service, the CCM calls the corresponding delete API to tear down the cloud resource. This tight coupling means orphaned load balancers can occur if the CCM is down or misconfigured during deletion.

Cloud-Specific Annotations

Kubernetes provides the type: LoadBalancer abstraction, but cloud providers expose dozens of configuration options that do not map to any standard Kubernetes field. These options are surfaced through annotations on the Service metadata.

AWS: NLB vs Classic Load Balancer

By default, AWS provisions a Classic Load Balancer (CLB). To use a Network Load Balancer (NLB), which operates at Layer 4 and supports static IPs, higher throughput, and lower latency, set:

metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

The AWS Load Balancer Controller (a newer, separate controller) uses a different annotation namespace (service.beta.kubernetes.io/aws-load-balancer-nlb-target-type) and supports additional features like IP-mode targets, which route directly to pod IPs rather than through NodePorts.

GCP

GCP defaults to an external TCP/UDP load balancer. For an internal LB, annotate:

metadata:
  annotations:
    networking.gke.io/load-balancer-type: "Internal"

Azure

Azure uses a similar pattern. For an internal LB:

metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

MetalLB for On-Premises Clusters

In bare-metal or on-premises environments, there is no cloud controller manager and no cloud API to provision load balancers. MetalLB is the standard solution. It runs as a set of pods in your cluster and implements the LoadBalancer Service type by announcing IP addresses to the local network.

L2 Mode (ARP/NDP)

In L2 mode, one MetalLB speaker pod claims the allocated IP by responding to ARP requests on the local network. This is the simplest setup -- it works on any Layer 2 network without router configuration. However, it has a limitation: only one node handles all traffic for a given IP. If that node fails, MetalLB fails over to another node (typically within 10 seconds), but during normal operation there is no multi-node load distribution at the network level.

BGP Mode

In BGP mode, MetalLB peers with your network routers and announces IP routes via BGP. This allows routers to distribute traffic across multiple nodes using ECMP (Equal-Cost Multi-Path). BGP mode provides true high availability and load distribution, but requires BGP-capable routers and network team coordination.

Configuration

MetalLB uses two custom resources: IPAddressPool defines the range of IPs available for allocation, and L2Advertisement or BGPAdvertisement controls how those IPs are announced.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: production-pool
  namespace: metallb-system
spec:
  addresses:
    - 192.168.1.200-192.168.1.250
    - 10.0.0.100/32          # single IP also works
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-announce
  namespace: metallb-system
spec:
  ipAddressPools:
    - production-pool

Internal vs External Load Balancers

Not every service should be reachable from the internet. Internal load balancers are provisioned inside your VPC/VNET and only accessible from within the private network. This is the correct choice for backend services, databases, and inter-service communication that should never be exposed publicly.

The distinction is controlled entirely through annotations (shown in the cloud-specific section above). From Kubernetes' perspective, both are type: LoadBalancer -- the annotation tells the CCM whether to create a public or private resource.

Health Check Configuration

Cloud load balancers perform their own health checks, independent of Kubernetes readiness and liveness probes. If a node fails the LB health check, the cloud LB stops sending traffic to it. These checks are configured via annotations and are cloud-specific.

Common parameters include the check interval (how often to probe), the timeout (how long to wait for a response), and thresholds (how many consecutive successes/failures to require before changing state). For HTTP health checks, you also specify the path and expected response code.

Session Affinity (Sticky Sessions)

By default, Kubernetes distributes requests across all ready pods. For stateful applications -- websocket connections, file uploads, shopping carts -- you may need all requests from a given client to reach the same pod.

Kubernetes supports sessionAffinity: ClientIP at the Service level. When enabled, kube-proxy uses the client's source IP to consistently route to the same backend pod for a configurable timeout period (default: 10800 seconds / 3 hours).

spec:
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 3600

Note that this works at the Kubernetes level (kube-proxy/iptables/IPVS). Cloud LBs may also offer their own sticky session mechanisms (e.g., cookie-based affinity on ALB), which operate independently.

externalTrafficPolicy

The externalTrafficPolicy field on a Service controls whether external traffic is routed only to pods on the node that received it, or can be forwarded cluster-wide.

• Cluster (default): Traffic arriving at any node is forwarded to any pod, regardless of which node hosts it. This provides even distribution but performs SNAT, which replaces the client's real IP with the node's internal IP.
• Local: Traffic is only sent to pods on the same node that received it. If no pod exists on that node, the traffic is dropped (and the LB health check should catch this). This preserves the client's original source IP, which is critical for logging, geo-targeting, and rate limiting.

Static vs Dynamic IP Allocation

By default, cloud LBs receive a dynamically allocated IP. If the LB is deleted and recreated, it gets a new IP, which means DNS records must be updated. For production services, a static IP is preferable.

On AWS, you can associate Elastic IPs with an NLB. On GCP, you reserve a static IP and reference it in spec.loadBalancerIP. On Azure, you create a public IP resource and reference it via annotations. In all cases, the static IP survives LB recreation.

spec:
  type: LoadBalancer
  loadBalancerIP: 35.200.10.50

Templating LB Annotations in Helm

A well-designed Helm chart should work across cloud providers without code changes. The standard pattern is to expose annotations as a map in values.yaml and merge them into the Service template:

# values.yaml
service:
  type: LoadBalancer
  port: 80
  annotations: {}
  loadBalancerIP: ""
  externalTrafficPolicy: Cluster
  sessionAffinity: None

# templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: {{ include "mychart.fullname" . }}
  {{- with .Values.service.annotations }}
  annotations:
    {{- toYaml . | nindent 4 }}
  {{- end }}
spec:
  type: {{ .Values.service.type }}
  externalTrafficPolicy: {{ .Values.service.externalTrafficPolicy }}
  sessionAffinity: {{ .Values.service.sessionAffinity }}
  {{- if .Values.service.loadBalancerIP }}
  loadBalancerIP: {{ .Values.service.loadBalancerIP }}
  {{- end }}
  ports:
    - port: {{ .Values.service.port }}
      targetPort: http
  selector:
    {{- include "mychart.selectorLabels" . | nindent 4 }}

Users then override annotations per environment:

# AWS production
helm install myapp ./mychart -f values-aws-prod.yaml

# On-prem with MetalLB
helm install myapp ./mychart --set service.loadBalancerIP=192.168.1.200

Cost Considerations

Every LoadBalancer Service provisions a separate cloud load balancer. On AWS, an NLB costs approximately $16/month in fixed charges plus data processing fees. If you have 20 microservices each exposed as a LoadBalancer, you are paying for 20 NLBs -- over $300/month before data charges.

The alternative is a shared Ingress controller: deploy one LoadBalancer Service for the ingress controller (e.g., NGINX or Traefik), then use Ingress resources for L7 routing to your services. This collapses 20 LBs into one, saving significant cost. The trade-off is added complexity (TLS termination, routing rules, shared failure domain) and the fact that Ingress operates at Layer 7 only -- if you need raw TCP/UDP load balancing, you still need individual LoadBalancer Services.

Further Reading

• Kubernetes Docs -- Service type LoadBalancer
• MetalLB Official Documentation
• AWS Load Balancer Controller
• Creating an External Load Balancer