Spot interruptions are a scheduling problem

Wed, 28 Oct 2026 00:00:00 GMT

Most spot horror stories have the same shape. A node disappears, a few pods with no disruption budget go with it, the remaining replicas take the full load, and someone gets paged for a latency spike that resolves itself four minutes later. The postmortem blames spot. The real cause was a cluster that assumed nodes live forever.

EC2 gives you a two-minute warning before it reclaims a Spot instance. That is plenty of time to reschedule a stateless pod. Whether it is enough depends on what you did before the warning arrived. This is the checklist I use, in the order I apply it.

#The two-minute notice

When AWS needs the capacity back, it publishes a Spot Instance Interruption Warning to EventBridge and the instance metadata service, then reclaims the instance about 120 seconds later. Karpenter can consume these events from an SQS queue. With interruption handling on, it taints and drains the node as soon as the warning lands, and launches replacement capacity in parallel instead of waiting for pods to go Pending.¹

Figure 1One interruption, to scale. Durations are illustrative, not measured. The shaded window is the time between the old pods leaving and the new ones becoming ready.

Two things follow from that timeline. Replacement capacity takes time to boot, so there is usually a window where the surviving replicas carry the load alone. Size for it. And every pod on the node has to finish shutting down inside the notice, so a 300-second termination grace period is a promise you can’t keep.

#Diversify instance types

Interruption rates are per capacity pool: one instance type in one Availability Zone. The more pools a NodePool can draw from, the more likely the allocation strategy finds one with spare capacity, and the less likely a single reclaim event takes several of your nodes at once.

The common mistake is pinning instance types. Constrain on what the workload actually needs (architecture, generation, a size range) and let everything that fits in.

nodepool.yaml

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-spot
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: karpenter.k8s.aws/instance-cpu
          operator: In
          values: ["4", "8", "16", "32"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m

Allowing arm64 roughly doubles the pool count, but only if every image the pool might run is multi-arch. Check before you add it. Listing both capacity types lets Karpenter fall back to on-demand when no spot offering fits, instead of leaving pods Pending.²

#Make the drain boring

A drain is only as graceful as the pods on the node. Three settings decide whether an interruption is invisible or a page.

Setting	What it protects	Where I start
`PodDisruptionBudget`	Pods leave one at a time while replacements come up elsewhere	`maxUnavailable: 1`, or 10–25% for large deployments
`preStop` sleep	The load balancer stops sending new requests before the process exits	10–15 s, longer than the target’s deregistration lag
`terminationGracePeriodSeconds`	Total time for the preStop hook plus graceful shutdown	Under 90 s, so it fits inside the notice

A PodDisruptionBudget can’t stop AWS from taking the instance back. What it does is pace the drain. Keep the budget loose enough that the drain finishes inside the window, or the last pods are killed instead of evicted.

deployment.yaml (excerpt)

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: api
          lifecycle:
            preStop:
              sleep:
                seconds: 15
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: api }
        - maxSkew: 1
          topologyKey: karpenter.sh/capacity-type
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: api }

The native sleep action saves you from shipping a shell in the image just to wait.³ The second spread constraint is the one people skip, and it matters most in the next section.

#Keep an on-demand floor

Spot reclaims come in waves. When a pool gets tight, several instances in it go at once. Spreading replicas across capacity types means a wave can never take all of them. With karpenter.sh/capacity-type as a topology key and maxSkew: 1, Karpenter launches a mix of spot and on-demand nodes to satisfy the constraint.

For services where a cold start hurts, I add a small on-demand NodePool with a higher weight, sized to hold the minimum replica count. It costs a little. It also means the worst case is degraded, not down.

#What I watch

An interruption should show up on a dashboard, not in a page. These are the signals that tell me the setup is working:

Interruption events, as annotations. Overlay them on the service’s latency and error panels. If the lines don’t move when an annotation appears, you’re done.
Pending time after an interruption. How long evicted pods wait for a node. If it grows, the NodePool is too narrow.
Fallback rate. The share of nodes launched as on-demand. A rising rate means spot capacity in your pools is getting scarce, and your savings with it.

bash

# Interruption warnings Karpenter acted on recently
kubectl get events -A --field-selector reason=SpotInterrupted

# Current nodes with capacity type and zone
kubectl get nodes -L karpenter.sh/capacity-type,topology.kubernetes.io/zone

#What stays off spot

Anything that can’t lose its node in two minutes: StatefulSets without fast replica promotion, long batch jobs without checkpoints, and the cluster’s own controllers. Karpenter itself should run on capacity it doesn’t manage, such as a small managed node group or Fargate. Otherwise an interruption can take out the thing that handles interruptions.

None of this is specific to spot. Nodes fail for plenty of reasons, and a cluster that handles a two-minute warning gracefully handles a kernel panic better too. Spot just makes you prove it every week.

Part 2 covers the cost side: how to measure what spot is actually saving once you count the on-demand fallback and the extra headroom.

Notes

The same queue also carries scheduled maintenance events and instance state changes, which Karpenter handles the same way: cordon, drain, replace. ↩
When a NodePool allows both, Karpenter prefers spot and uses on-demand only when no spot offering satisfies the pod’s requirements. ↩
The sleep lifecycle action is on by default from Kubernetes 1.30. On older clusters, use an exec hook that runs sleep from the image. ↩

Ricardo Herrera · Writing

Spot interruptions are a scheduling problem

#The two-minute notice

#Diversify instance types

#Make the drain boring

#Keep an on-demand floor

#What I watch

#What stays off spot

Notes