Cost optimization in AKS: maxPods, the invisible bottleneck

The problem: 3 idle nodes that don't scale down

You receive a cost-optimization ticket. The scenario is classic:

• An AKS with 3 nodes in the node pool (configured as 1-3 with autoscaler)
• Only 2 real application pods (app-water and app-corporate)
• 23 operations pods from the cluster itself
• CPU at 8%, memory at 12%
• Nobody understands why there are 3 active nodes if there is no load

The request is clear: "We need to size the machines to reality. We want 1 node. If there are limitations due to the number of pods, let's review what can be removed."

Intuition says: "If there is barely any load, why does Kubernetes keep extra nodes?" The answer is not in the CPU nor in the RAM. It is in a parameter almost nobody reviews: maxPods.

Anatomy of an AKS node: the invisible pods

Before diagnosing, you need to understand that in AKS each node comes "preloaded" with system components that consume pod slots. A typical node with Azure CNI, monitoring and security policies has this distribution:

Untouchable pods (Kubernetes and Azure core)

That is already 11 pods that you cannot touch. And each one consumes a slot of the maxPods limit.

Governance and security pods (reviewable)

Observability pods (the suspects)

Step-by-step diagnosis

Step 1: Count pods per node

kubectl get pods --all-namespaces -o wide | awk '{print $8}' | sort | uniq -c

Result:

     26 aks-nodepool-xxxxx-vmss000004
     28 aks-nodepool-xxxxx-vmss00001c

Between the two nodes they add up to 54 pods. If the per-node limit is 30, it is mathematically impossible to consolidate into a single one.

Step 2: Confirm the node pool configuration

az aks nodepool show \
  --resource-group DEV-RG-AKS-01 \
  --cluster-name DEV-AKS-01 \
  --name defaultv2 \
  --query "{minCount: minCount, maxCount: maxCount, count: count, enableAutoScaling: enableAutoScaling, maxPods: maxPods}"

Result:

{
  "count": 2,
  "enableAutoScaling": true,
  "maxCount": 2,
  "maxPods": 30,
  "minCount": 1
}

There is the problem: maxPods: 30. The Cluster Autoscaler calculates: "If I remove node 2, I need to relocate 28 pods onto node 1. But node 1 only has 4 free slots (30 - 26). Impossible. I keep 2 nodes."

Step 3: Check resources (CPU/Memory)

kubectl top nodes

NAME                                CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
aks-nodepool-xxxxx-vmss000004       346m         8%     4053Mi          12%
aks-nodepool-xxxxx-vmss00001c       257m         6%     3701Mi          11%

Combined: 14% CPU, 23% Memory. A single node has plenty of capacity to host the entire load. The bottleneck is not hardware, it is the IP/pod limit.

Step 4: Check available IPs in the subnet

In Azure CNI, each pod reserves a private IP from the subnet. If you raise maxPods to 60, the node will need 60 IPs.

Check the available space in the Azure portal: Virtual Network → Subnet → Available IPs.

In our case, the /23 subnet has 512 total IPs (~400 available). More than enough space.

Why is maxPods 30 by default?

Azure sets 30 as the default value in Azure CNI out of conservatism:

• Each pod consumes a real IP from the subnet
• In small subnets (/26, /27), a high maxPods would exhaust the IPs
• Microsoft prefers that the cluster "works" with the most restrictive config

The problem is that this default is carried over from cluster creation and nobody reviews it. In a /23 subnet with 512 IPs, keeping maxPods at 30 is like buying a 500-space parking lot and putting up a "maximum 30 cars" sign.

The solution: cleanup + node pool recreation

Can you just clean up pods to get down to 30?

Let's do the math applying the maximum cleanup:

Result: 46 minimum pods. They still don't fit on a node with maxPods=30.

The base components of Azure CNI, CSI drivers and Kubernetes core are immutable. You cannot go below ~42 pods on an AKS node with the standard corporate stack.

The real solution: recreate the node pool with maxPods=60

The maxPods parameter is locked by Azure — it cannot be modified live on an existing node pool. The only option is a Blue/Green migration:

Phase 1: Prior cleanup

# Reduce ingress replicas to 1 (INT/DES does not require HA)
kubectl scale deployment clippy --replicas=1 -n ingress-basic
kubectl scale deployment ingress-nginx-controller --replicas=1 -n ingress-basic

# Disable Azure Monitor (removes ama-* pods)
az aks disable-addons --addons monitoring \
  --name INT-AKS-01 \
  --resource-group INT-RG-AKS-01

# Remove Prometheus (if deployed via Helm)
helm uninstall prometheus-stack -n prometheus

Phase 2: Create the new node pool

# Create new pool with maxPods=60 and 1 fixed node
az aks nodepool add \
  --resource-group DEV-RG-AKS-01 \
  --cluster-name DEV-AKS-01 \
  --name newpool \
  --node-count 1 \
  --max-pods 60 \
  --node-vm-size Standard_D4s_v5 \
  --mode System \
  --no-wait

Note: Use --mode System so it can host critical system pods. Verify the VM size with az aks nodepool show on the current pool.

Phase 3: Migrate pods to the new node

# Block scheduling on the old pool
kubectl cordon -l agentpool=defaultv2

# Evict pods (they will be relocated to the new node)
kubectl drain -l agentpool=defaultv2 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60

Phase 4: Verify and delete the old pool

# Verify that all pods are on the new node
kubectl get pods --all-namespaces -o wide | awk '{print $8}' | sort | uniq -c

# If everything is OK, delete the old pool
az aks nodepool delete \
  --resource-group DEV-RG-AKS-01 \
  --cluster-name DEV-AKS-01 \
  --name defaultv2 \
  --no-wait

Phase 5: Rename (optional)

If you need to keep the corporate naming:

# Create the definitive pool with the correct name
az aks nodepool add \
  --resource-group DEV-RG-AKS-01 \
  --cluster-name DEV-AKS-01 \
  --name defaultv2 \
  --node-count 1 \
  --max-pods 60 \
  --node-vm-size Standard_D4s_v5 \
  --mode System

# Migrate from newpool
kubectl cordon -l agentpool=newpool
kubectl drain -l agentpool=newpool --ignore-daemonsets --delete-emptydir-data

# Delete the temporary pool
az aks nodepool delete \
  --resource-group DEV-RG-AKS-01 \
  --cluster-name DEV-AKS-01 \
  --name newpool

Final result

After the migration:

$ kubectl get nodes
NAME                              STATUS   ROLES    AGE   VERSION
aks-defaultv2-xxxxx-vmss000000    Ready    <none>   2h    v1.35.4

$ kubectl get pods --all-namespaces | wc -l
47

$ kubectl top nodes
NAME                              CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
aks-defaultv2-xxxxx-vmss000000    603m         15%    7754Mi          24%

A single node with 46 pods out of the 60 allowed, CPU at 15% and memory at 24%. The second and third nodes no longer exist and stop billing.

Decision matrix: what to remove in DEV

For non-production environments, this is the guide to what can be trimmed:

Lessons learned

1. maxPods is the real limiter, not the CPU nor the RAM. Review it every time you create a new cluster.

1. The default of 30 is insufficient for any environment with corporate monitoring. Recommendation: always configure 60-110 from day 0.

1. Azure CNI consumes IPs in advance — check your subnet space before raising maxPods. Formula: nodes × maxPods ≤ available IPs in subnet.

1. The Cluster Autoscaler cannot violate maxPods — even if you have minCount=1, if the pods don't fit on a single node, the autoscaler will keep extra nodes.

1. In DEV/DES you don't need HA — 3 ingress replicas, 2 nginx controllers and corporate backup in a test environment is money thrown away.

1. Recreating the node pool is the only option to change maxPods. Plan a maintenance window and use the Blue/Green strategy.

Feasibility checklist before executing

Before proposing this solution to your team, verify these 3 points:

• [ ] Available IPs: subnet_size - 5 (Azure reserved) - (current_nodes × current_maxPods) > new_maxPods
• [ ] CPU/Memory: Sum of kubectl top nodes < 70% on a single node of the chosen size
• [ ] Stateful applications: If there are zonal PersistentVolumes, the new node must be in the same availability zone

If all three are affirmative, the migration is safe and without loss of service.

References

• Azure AKS - Configure max pods per node
• Cluster Autoscaler - Scale down limitations
• Azure CNI IP planning
• Node pool migration strategies