How to Handle your Kubernetes Upgrades
Download as PPTX, PDF1 like478 views
This document provides a comprehensive guide on upgrading Kubernetes clusters, detailing methodologies, an update flowchart, and potential issues encountered during the process. Key points include a hop-by-hop upgrade strategy, specific command instructions for updating various components, and troubleshooting common problems. Additionally, resources for further information and automation ideas for streamlining the upgrade process are included.
![$ ls -ltr
• Architecture and General methodology
• Problems/Errors faced
• Update Flowchart
• Resources used for help
• Heads-up for things to look for when upgrading!
• [ bonus slide ;) ] Automation ideas
• Questions?
@_suvzz](https://image.slidesharecdn.com/suvrojeetgoshhandleupgrades-190307184023/85/How-to-Handle-your-Kubernetes-Upgrades-3-320.jpg)
![Heads-up for things to look for when
upgrading
• Change the version on top right of kubernetes.io and then manually find the
document you need because if you search it will always point you to the latest
version. Docs older than v1.9 are not available
• Always check the Release notes/Changelog before performing any upgrade
operation. Watch out for “[action required]” and/or “Before Upgrading”
• Features gate changed from string “” to map {“”} PR:57962 in kubeadm-config.
Applicable upgrading from v1.9 to v1.10
• (version v1.10 to v1.11) there has been change in kubeadm config file format
MasterConfiguration (v1alpha2) is broken down (in v1alpha3)to InitConfiguration
and ClusterConfiguration. Source
• (version v1.10 to v1.11) Major changes in the drop-in file
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf. Keep an eye here and put in your
EXTRA KUBELET ARGS accordingly to the file it belongs. Source
• (version v1.11 to v1.12) control plane images don’t require architecture type
anymore. Source
@_suvzz](https://image.slidesharecdn.com/suvrojeetgoshhandleupgrades-190307184023/85/How-to-Handle-your-Kubernetes-Upgrades-10-320.jpg)
How to Handle your Kubernetes Upgrades
- 1. How to Handle your Upgrades Thursday, March 7, 2019
- 2. $ whoami - Suvro $ kubectl describe suvro Suvrojeet Kumar Ghosh [email protected] by day at Ribbon Communications and by evening GitHub: @fOO223Fr Twitter: @_suvzz
- 3. $ ls -ltr • Architecture and General methodology • Problems/Errors faced • Update Flowchart • Resources used for help • Heads-up for things to look for when upgrading! • [ bonus slide ;) ] Automation ideas • Questions? @_suvzz
- 4. Architecture Master Master Worker Worker Worker VIP passiveactive GlusterFS for etcd BackU p @_suvzz
- 5. General methodology • Hop by Hop • example: v1.9.2 to v1.12.5 v1.9. 2 v1.10. 12 v1.11. 0 Reason for such a kind of methodology are two: 1. Usually the patch 0 of every version has major changes from the last lower version. 2. The last path version (incase of v1.10 is 12) is kind of the accumulation of the intermediate patches. So before moving next major version running though this has proved beneficial for many. Note: this is a strategy and not everyone need to follow it because it depends what features your cluster is using and what has changed. But by far most are benefitted following this strategy. @_suvzz
- 6. Update Flowchart Update kubeadm binary kubeadm config images list kubeadm config images pull kubeadm upgrade apply v1.XX.x kubeadm config migrate Update your CNI Update all binaries kubectl , kubeadm, kubelet Optional for newer version > v1.9 Version older than v1.9 download a temp kubeadm binary, else it might mess with the drop in file. 10-kubeadm.conf Version > v1.11.0, take advantage of these cool subcommands helps if you have a private docker repo then you can download ahead of upgrade and tag it appropriately. I have used this ever since v1.11.0 saves a lot of time in replacing deprecated flags or configuration changes. kubeadm upgrade plan Check and/or Update nodeName to current Master kubectl edit configmap -n kube- system kubeadm-config -o yaml @_suvzz
- 7. Flowchart continued.. (optional) Update kubectl in worker nodes kubectl drain $NODEs --ignore- daemonsets Update kubelet version in worker nodes kubeadm upgrade node config --kubelet- version v1.X.x Update the kubeadm config map Restart kubelet Kubectl uncordon $node @_suvzz
- 8. Problems • Backing off of containers! • Etcd: possible errors pid locked by last etcd and/or deprecated API endpoints. • Apiserver: usually when etcd unhealthy and/or deprecated flags. • Mirror pod hash race condition between etcd and apiserver which make upgrade to fail in the older versions. PR:61942 I have faced this problem in v1.9 and was fixed with retrying couple of times. • (Version < v1.11 )if kubelet fails to understand the node ip or if you are using private IP, use –-node-ip flag to explicitly mention the IP. • Explicitly mention --cgroup-driver to kubelet (in my case: cgroupfs but was detected as systemd) @_suvzz
- 9. Resources • Godoc: https://godoc.org/k8s.io/kubernetes (I use this often to check on struct and value types) • CHANGELOGs: https://github.com/kubernetes/kubernetes/CHANGELOG- 1.X.X.md (Holy grail for updates) • Upgrade docs in kubernetes.io: https://kubernetes.io/docs/tasks/administer- cluster/kubeadm/ @_suvzz
- 10. Heads-up for things to look for when upgrading • Change the version on top right of kubernetes.io and then manually find the document you need because if you search it will always point you to the latest version. Docs older than v1.9 are not available • Always check the Release notes/Changelog before performing any upgrade operation. Watch out for “[action required]” and/or “Before Upgrading” • Features gate changed from string “” to map {“”} PR:57962 in kubeadm-config. Applicable upgrading from v1.9 to v1.10 • (version v1.10 to v1.11) there has been change in kubeadm config file format MasterConfiguration (v1alpha2) is broken down (in v1alpha3)to InitConfiguration and ClusterConfiguration. Source • (version v1.10 to v1.11) Major changes in the drop-in file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf. Keep an eye here and put in your EXTRA KUBELET ARGS accordingly to the file it belongs. Source • (version v1.11 to v1.12) control plane images don’t require architecture type anymore. Source @_suvzz
- 11. Automation ideas • Ansible roles • Warmup role • Changes with every major version of k8s. All the release notes stuff goes in here. • Update role • Common module for all updates. This role runs after the warmup role • Bash helper scripts • script to drain nodes • Script to cordon and uncordon nodes @_suvzz
- 12. $ poweroff • The upgrade process is complicated but it worth taking the journey and moving to the latest stable version to enjoy all the cool Graduated stuffs! • Upgrade has taught me a lot about kubernetes. @_suvzz Source