Reading about cluster upgrades teaches you vocabulary; rehearsing one teaches you judgement. This tutorial walks through a rehearsal you can run in a lab, built around the sequence described in the Atlas entry on KubernetesOps from xOps School. The aim is a written, repeatable runbook for a single minor-version hop, and a feel for the places where upgrades go sideways.
Before you start: the lab setup
Prepare a disposable cluster that mirrors production in shape, if not in size. The Atlas suggests kind for labs, which creates clusters inside containers on one machine. For a more realistic rehearsal use two or three virtual machines with the standard installer. Pick a starting version exactly one minor behind the target, because skipping minors is not allowed on the API server.
Step 1: Read the removal notes
Open the release notes for the target minor and list everything labelled deprecated or removed. Then search your manifests for each item. The Atlas gives a cautionary tale: a label removal that the project had flagged beforehand nonetheless broke routing at a large social platform during a 1.23 to 1.24 hop in March 2023, because nobody searched for it. On a live cluster there is also a counter of deprecated calls exposed by the API server. It resets whenever the process restarts, so the deprecation annotations in audit logs give you a longer trail.
Step 2: Check health and skew
Confirm every node is ready, no pods are stuck pending and nothing is crash-looping. Then write down the versions of each component. Node agents must not be newer than the API server and can be up to three minors older. This is also the moment to look for budgets that allow zero disruption, since they will stall Step 5.
Step 3: Take and test a datastore snapshot
Save an etcd snapshot, record its size and location, and restore it into a scratch cluster to prove it works. Guard the file closely: per the Atlas, whoever can read the datastore effectively holds root on the cluster. If your lab restore fails, you have just found a bug cheaply. Record the duration, since an honest restore time is a number you will want later.
Step 4: Upgrade the control plane
Work through the components in the documented order. The API server goes first. The controller manager, scheduler and cloud controller follow, and the node agents and network proxy come after. With the standard installer, planning and applying the upgrade on the first control-plane node is followed by an upgrade of the remaining control-plane nodes. The Atlas notes the tool can safely be re-run if interrupted, with backups of files it changed placed under a temporary directory.
After each component, check that the API responds and that system pods return to ready. Do not continue while anything looks wrong.
Step 5: Drain, upgrade and return each worker
- Cordon and drain one node. The drain uses the eviction mechanism, so budgets are respected.
- If evictions are refused, wait and retry; the refusal is a rate-limit style response. If you see a server error, look for two budgets selecting one pod.
- Upgrade the node agent and restart it.
- Uncordon the node and watch workloads settle.
- Repeat for the next node only after the previous one is healthy.
Resist force-deleting a pod to get past a stubborn drain. It bypasses the protection on purpose, and the Atlas keeps it for emergencies agreed with the owner of the workload.
Step 6: Verify and measure
Run your smoke tests, then compare the post-upgrade state with Step 2. Check that add-ons such as DNS and the network plugin run versions compatible with the new minor. Record the elapsed time: this is your upgrade lead time, one of the measures the Atlas recommends tracking over several quarters. Also run the certificate expiry command to see how far away renewal is; client certificates from the installer last a year, so a date within a few months deserves a calendar entry.
Step 7: Write the runbook while it is fresh
Turn your terminal history into numbered instructions with expected outputs, the pitfalls you met and the decision points. The Atlas places “two consecutive hops from a runbook” at its repeatable maturity level, which is a useful target: if a colleague can follow your document to upgrade the next minor without asking you questions, the document is good enough.
Step 8: Repeat with a twist
Run the exercise again, deliberately introducing one fault each time: a zero-disruption budget, a liveness probe that calls a database, a webhook that fails closed with nobody home. The Atlas notes that webhooks reject requests by default when unreachable, so keep them out of the system namespace and the network plugin. Learning how each fault looks in the logs is worth more than a flawless first run.
Where this fits in a career
The CKA’s heaviest domain is troubleshooting at 30 percent, and this lab touches it repeatedly. Capture each fault you create and how you diagnosed it; that notebook becomes interview material. To build the diagnostic mindset more generally, our guide on building critical thinking for operational problem solving pairs well with the exercise, and for the larger picture of how such skills feed into change in an organisation, see the skills roadmap behind IT transformation work.
When you finish, you should own three artefacts: a tested snapshot procedure, a runbook for one hop and a list of faults you can now recognise. That is a small body of evidence, but it shows the thing employers look for, which is calm competence under a changing system.
