Kubernetes Requests and Limits: QoS, Quotas and Right-Sizing

A cluster can refuse to schedule anything while its nodes sit at 15% actual load, and no amount of extra capacity fixes it because the constraint is arithmetic rather than hardware. The two numbers on every container drive that, along with eviction order and a throttling behaviour that produces latency spikes on idle machines. This covers how to derive them from measurement.

Two numbers per container decide how much of your cluster you can actually use. Requests are what the scheduler reserves; limits are what the kernel enforces. Setting them from a guess is why clusters run at 30% utilisation while refusing to schedule anything, and why the fix is usually to lower numbers rather than buy nodes.

Requests Schedule, Limits Enforce

The scheduler only ever looks at requests. It sums the requests of pods already on a node, compares against allocatable capacity, and places your pod where the sum still fits. Actual usage does not enter into it.

Limits are enforced at runtime by the kernel, and the two resources behave completely differently at the limit.

That asymmetry drives the main recommendation in this post. CPU is compressible, so a container starved of it slows down. Memory is not, so a container over its limit is destroyed. Treat the two numbers as different kinds of decision rather than a pair.

Our guide to pod troubleshooting covers diagnosing both from the other end, once something has already gone wrong.

The QoS Class You Did Not Know You Chose

Kubernetes derives a quality-of-service class from your requests and limits, and that class decides who gets evicted when a node runs short of memory. Nobody sets it directly and everybody has one.

Guaranteed: every container has requests equal to limits, for both CPU and memory. Evicted last. Also eligible for exclusive CPU pinning under the static CPU manager policy, which matters for latency-sensitive work.

Burstable: requests set, limits higher or absent. Evicted after BestEffort, in order of how far usage exceeds requests. The right class for most workloads.

BestEffort: nothing set at all. Evicted first, and its usage is invisible to the scheduler, so it is also the thing that makes a node fall over. Never ship this to production.

kubectl get pod my-app -o jsonpath='{.status.qosClass}'
kubectl get pods -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,QOS:.status.qosClass' \
  | grep BestEffort

Run that second command against your cluster now. The output is usually longer than expected, and every line is a pod the scheduler is placing blind.

Why CPU Limits Are Usually a Mistake

This is the least intuitive part and the change that most often makes a cluster faster.

CPU limits are enforced by CFS quota over a 100 ms period. A container with a 500 m limit gets 50 ms of CPU per period. Once spent, it is frozen until the next period begins, regardless of how idle the node is.

For a multi-threaded runtime this is worse than it sounds, because the quota is consumed across all threads. Eight threads burning 10 ms each exhaust a 50 ms quota in a fraction of the period, and then the whole container stalls. The result is p99 latency spikes on a machine with idle cores.

kubectl exec my-app -- cat /sys/fs/cgroup/cpu.stat | grep -E 'nr_throttled|throttled_usec'

A rising nr_throttled on a service with latency complaints is your answer. The usual fix is to set CPU requests accurately and remove the CPU limit, letting the pod use spare capacity while requests still guarantee its share under contention. Keep CPU limits where you genuinely need predictable ceilings, such as untrusted workloads or hard multi-tenancy.

Memory limits stay. Always. An unbounded container with a leak takes the node down and everything on it.

Getting the Numbers From Measurement

Guessing produces both failure modes at once: requests too high, so the cluster refuses work while sitting idle, and memory limits too low, so pods restart under normal peaks.

kubectl top pods -n prod --sort-by=memory
# with Prometheus, the numbers that matter:
#   memory:  max_over_time(container_memory_working_set_bytes[7d])
#   cpu p95: quantile_over_time(0.95, rate(container_cpu_usage_seconds_total[5m])[7d:])

Then a rule that holds up in practice. Set the memory request to observed peak working set and the memory limit 20 to 50% above it, high enough to survive a real spike. Set the CPU request to the p95 of actual usage rather than the peak, because CPU is compressible and reserving for peak wastes the cluster.

Use a week of data, not an hour, and make sure the week includes whatever your busy period is. Sizing from a quiet Sunday produces limits that fail on Monday. Our walkthrough for Prometheus and Grafana covers collecting the history these queries need.

LimitRange and ResourceQuota

Two namespace-scoped objects, frequently confused, doing opposite jobs.

LimitRange applies defaults and bounds to individual containers. Its most useful property is supplying defaults, which turns a BestEffort pod from a careless team into a Burstable one automatically.

apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: prod }
spec:
  limits:
    - type: Container
      default:        { memory: 512Mi }
      defaultRequest: { cpu: 100m, memory: 256Mi }
      max:            { cpu: "4", memory: 8Gi }

ResourceQuota caps the namespace in aggregate, so one team cannot consume the cluster. Note the trap: once a quota specifies a resource, every pod in that namespace must set that resource or be rejected outright. Pair a quota with a LimitRange supplying defaults, or you will break every deployment that omits a value.

apiVersion: v1
kind: ResourceQuota
metadata: { name: team-cap, namespace: prod }
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.memory: 64Gi
    persistentvolumeclaims: "20"

Before You Add Nodes

When pods will not schedule, compare requested against allocatable before assuming the cluster is full.

kubectl describe node worker-2 | sed -n '/Allocated resources/,/Events/p'

A node showing 95% of CPU requested and 15% actually in use is not short of capacity, it is short of accuracy. Fixing the requests on the three worst offenders frequently recovers more headroom than another node would, at no cost.

Two other things to check. Every node reserves capacity for the kubelet and the OS, so allocatable is meaningfully less than the machine's total, and a request sized to the full node will never schedule. And on a GPU node, a card is requested integrally and held exclusively whether or not it is used, which is the most expensive version of this problem: our guide to GPU monitoring covers catching an allocated but idle card.

The Capacity You Are Buying

Right-sizing is worth more when capacity is bought in small increments, because the alternative to a careful number is rounding up to the next instance size.

MassiveGRID's managed Kubernetes bills in cloudlets of 128 MiB of RAM and 400 MHz of CPU from $0.03474 per hour, about $25.37 a month, so a pod sized at 384 MiB is charged as three cloudlets rather than as a fraction of an instance you had to buy whole. That granularity is what makes measuring worthwhile. Underneath, Proxmox high-availability clustering with automatic failover over Ceph storage replicating every block three times across independent NVMe drives means a node lost to memory pressure is a restart elsewhere rather than an outage.

Clusters can be ordered across a partner footprint of more than 700 datacenters in 85 metros, 30 countries and six continents, with auto-provisioning in New York, London, Frankfurt and Singapore.