Spot VM reclamation taking workloads down during cost optimisation

How to confirm it

• Which PDBs would block a drain?

  kubectl get pdb -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed | awk '$3==0'

  Zero allowed disruptions means a graceful drain cannot proceed and the node hangs until it is force-deleted.

• Was this a preemption or a crash?

  kubectl get events -A --field-selector reason=Preempting,reason=NodeNotReady --sort-by=.lastTimestamp | tail -20

  Distinguishes 'the provider reclaimed capacity as agreed' from 'something actually broke'.

• How much is on Spot right now?

  kubectl get nodes -L cloud.google.com/gke-provisioning -o custom-columns=NODE:.metadata.name,TYPE:.metadata.labels.'cloud\.google\.com/gke-provisioning' | sort -k2

  If a critical workload has every replica on Spot nodes, a single reclamation event takes it out entirely.

• Is termination handled?

  kubectl get deploy -A -o json | jq -r '.items[] | select(.spec.template.spec.terminationGracePeriodSeconds > 25) | .metadata.name'

  Spot gives roughly 30 seconds. A grace period longer than that means the pod is killed mid-shutdown.