The problem with cost work
Every cloud bill has obvious waste in it, and every engineer knows roughly where. The reason it survives is not ignorance — it is that the cheap option is usually the less available one, and nobody wants to be the person who saved 30% and caused an outage. So the waste stays. Any cost programme that does not answer the availability question first will be reversed the moment something breaks, and the second attempt is much harder to get approved than the first.
Step 1 — separate the baseline from the spiky part
Before touching anything, split the workload in two. There is a floor of demand that exists at 3am on a Sunday, and there is everything above it. The floor should never be bought at on-demand prices; the spiky part should never be bought on a one-year commitment. Getting this split wrong in either direction is how cost programmes either save nothing or lock the company into capacity it stops needing in month three.
gcloud compute instances list --format='table(name,machineType.basename(),scheduling.provisioningModel,status)' | sort -k2Start with what is actually running, grouped by machine type and provisioning model. The gap between this and the recommender's view is where the money is.
Step 2 — commit only to the floor
Committed use discounts trade flexibility for roughly 30-55% off, depending on term and commitment type. They are the right instrument for the floor and the wrong one for anything else. The failure mode is committing to peak because peak is what the dashboard shows — you then pay for that capacity for a year whether or not you use it.
gcloud recommender recommendations list --recommender=google.compute.commitment.UsageCommitmentRecommender --location=asia-south1 --format=json | jq '.[].content.overview'GCP's own recommender models this from your last 30 days. Treat it as a starting position, not an answer — it does not know what you are about to migrate or decommission.
Step 3 — the failover has to land before the Spot migration
This is the part people invert, and it is why cost programmes get rolled back. Spot capacity is reclaimed with roughly 30 seconds of notice; that is the deal, not a defect. If a reclamation is an incident, you have not finished the work. Pod disruption budgets, sane termination grace periods, a handler that drains on the preemption signal, and an on-demand pool to fall back into — all of that ships first. Only then does moving workloads to Spot become a cost decision rather than a bet.
kubectl get pdb -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,MIN:.spec.minAvailable,ALLOWED:.status.disruptionsAllowed | awk '$4==0'Any PDB showing zero allowed disruptions will block a graceful drain and turn a routine reclamation into a stuck node. Fix these before a single workload moves.
Step 4 — autoscale on utilisation, not on fear
Most requests are set once, by whoever wrote the manifest, using a number that felt safe. Actual utilisation is usually a fraction of it, and the gap is pure cost with no reliability benefit — an over-requested pod is not more available, it just packs worse. The fix is continuous rightsizing driven by live telemetry rather than a quarterly spreadsheet exercise.
kubectl top pods -A --sort-by=cpu | head -30
# then compare against requests:
kubectl get pods -A -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpuThe ratio between these two columns, across a few hundred pods, is the size of the prize. Anything requesting 10x what it uses is paying for air.
What actually moved the number
Roughly: commitments on the floor, Spot for everything preemption-tolerant, and rightsizing to close the requests-versus-usage gap. None of it is clever in isolation. The engineering was in the ordering — availability work first, so that every subsequent saving was safe to keep. The 40% held because nothing broke, and nothing broke because the failover existed before the savings did.