Overview of kubernetes network functions
1 like988 views
This document provides an overview of Kubernetes networking, including its functionalities, services, and challenges. It explains concepts like container networking, the role of the Container Network Interface (CNI), and how to implement Kubernetes networking effectively. Additionally, it addresses advanced networking scenarios including the use of hardware acceleration and the Multus CNI for multiple network interfaces in pods.
Overview of kubernetes network functions
- 2. COSCUP2018 x openSUSE.Asia GNOME.Asia I am Hung-Wei Chiu Co-organizer of SDNDS-TW Co-organizer of CNTUUG I love Linux Network/Kubernetes/SDN You can find me at: blog.hwchiu.com
- 3. COSCUP2018 x openSUSE.Asia GNOME.Asia Outline What network functions about k8s How does above function implement What is the challenge about k8s
- 4. COSCUP2018 x openSUSE.Asia GNOME.Asia What network function kubernetes providers?
- 5. COSCUP2018 x openSUSE.Asia GNOME.Asia What Container Network ○ Connectivity ○ DNS Kubernetes services
- 6. COSCUP2018 x openSUSE.Asia GNOME.Asia Do You Know How Container Works?
- 7. COSCUP2018 x openSUSE.Asia GNOME.Asia Containersvs.VMs Containers are isolated, but share OS and where appropriate bins/libraries
- 8. COSCUP2018 x openSUSE.Asia GNOME.Asia HowDockerWorks We know docker is isolated, but how does it works? Linux kernel support the Namespaces mechanisms to partition kernel resources to different processes
- 9. COSCUP2018 x openSUSE.Asia GNOME.Asia HowDockerWorks Mount namespaces IPC namespaces PID namespaces Network namespaces User namespaces UTS namespaces ○ Unix Time System
- 10. COSCUP2018 x openSUSE.Asia GNOME.Asia NetworkNamespace Isolate the network functions. Including the ○ Network interfaces ○ Routing rules ○ Netfilter (iptables)
- 12. COSCUP2018 x openSUSE.Asia GNOME.Asia docker0 Linux Host Linux Host
- 13. COSCUP2018 x openSUSE.Asia GNOME.Asia docker0 docker0 ns1 Linux Host Linux Host Linux Host
- 14. COSCUP2018 x openSUSE.Asia GNOME.Asia docker0 docker0 docker0 ns1 ns1 veth1 veth0 Linux Host Linux Host Linux Host Linux Host
- 15. COSCUP2018 x openSUSE.Asia GNOME.Asia docker0 docker0 docker0docker0 ns1 ns1ns1 vth1 vth0 veth eth0 Linux Host Linux Host Linux Host Linux HostLinux Host
- 16. COSCUP2018 x openSUSE.Asia GNOME.Asia Before we talk about service, we must know why service exist.
- 17. COSCUP2018 x openSUSE.Asia GNOME.Asia Pods/Deployments We can deploy our applications as a containers in the kubernetes. There’re many kind of the container we can deploy ○ Pod ○ Deployment ○ Statefulset ○ DaemonSet
- 18. COSCUP2018 x openSUSE.Asia GNOME.Asia Deployment Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster Deployment: ○ Ngnix ○ Replica: 3 10.123.234.56 10.123.234.57 10.123.234.58
- 19. COSCUP2018 x openSUSE.Asia GNOME.Asia Access How we application access those Nginx servers? By IP address ○ 10.123.234.56:80 ○ 10.123.234.57:80 ○ 10.123.234.58:80 What’s the problem
- 20. COSCUP2018 x openSUSE.Asia GNOME.Asia Deployment Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster Deployment: ○ Ngnix ○ Replica: 3 10.123.234.56 10.123.234.57 10.123.234.58
- 21. COSCUP2018 x openSUSE.Asia GNOME.Asia Deployment Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster Deployment: ○ Ngnix ○ Replica: 3 10.123.234.56 10.123.234.57 10.123.234.75
- 22. COSCUP2018 x openSUSE.Asia GNOME.Asia Access How we application access those Nginx servers? By IP address ○ 10.123.234.56:80 ○ 10.123.234.57:80 ○ 10.123.234.58:80 ○ 10.123.234.75:80 It’s not easy for our application to handle those ip-changed situation.
- 23. COSCUP2018 x openSUSE.Asia GNOME.Asia The Service is used to solve this problem.
- 24. COSCUP2018 x openSUSE.Asia GNOME.Asia Service Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster 10.123.234.56 10.123.234.57 10.123.234.58 App Service Nginx
- 25. COSCUP2018 x openSUSE.Asia GNOME.Asia Service Application to Service ○ We use the DNS to access the service. ○ $(service).$(namespace).cluster.local Service to Pods ○ Service maintains all IP addresses of all Pods. ○ We call it endpoints
- 26. COSCUP2018 x openSUSE.Asia GNOME.Asia Service Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster 10.123.234.56 10.123.234.57 10.123.234.58 App Service Nginx nginx.default endpoints
- 27. COSCUP2018 x openSUSE.Asia GNOME.Asia How does kubernetes implements those functions?
- 28. COSCUP2018 x openSUSE.Asia GNOME.Asia What Container Network Connectivity ○ Container Network Interface(CNI) Kubernetes Services ○ There’re many implementation we can choose
- 30. COSCUP2018 x openSUSE.Asia GNOME.Asia ContainerNetworkInterface Cloud Native Computing Foundation Project Consists of a specification and libraries. Configure network interfaces in Linux containers Concerns itself only with network connectivity of containers ○ Create/Remove
- 31. COSCUP2018 x openSUSE.Asia GNOME.Asia ContainerNetworkInterface Removing allocated resources when the container is deleted
- 32. COSCUP2018 x openSUSE.Asia GNOME.Asia Who is using CNI?
- 33. COSCUP2018 x openSUSE.Asia GNOME.Asia FromtheGITHUB l rkt - container engine l Kubernetes - a system to simplify container operations l OpenShift - Kubernetes with additional enterprise features l Cloud Foundry - a platform for cloud applications l Apache Mesos - a distributed systems kernel l Amazon ECS - a highly scalable, high performance container management service
- 34. COSCUP2018 x openSUSE.Asia GNOME.Asia So, How to use the CNI?
- 35. COSCUP2018 x openSUSE.Asia GNOME.Asia StepbyStep Create a kubernetes cluster Setup your CNI plugin Deploy your first Pod
- 36. COSCUP2018 x openSUSE.Asia GNOME.Asia Just follow the installation to install the kubernetes
- 37. COSCUP2018 x openSUSE.Asia GNOME.Asia How do we install the CNI?
- 39. COSCUP2018 x openSUSE.Asia GNOME.Asia Handbyhand In the kubelet, we have the following parameters for CNI. --cni-bin-dir ○ /opt/cni/bin --cni-conf-dir ○ /etc/cni/net.d/ We should config the CNI for every k8s nodes.
- 42. COSCUP2018 x openSUSE.Asia GNOME.Asia Beforewestart Pod ○ A collection of containers
- 43. COSCUP2018 x openSUSE.Asia GNOME.Asia Steps Load the Pod config ○ Multiple containers Find a node to deploy the pod Create a Pause container
- 45. COSCUP2018 x openSUSE.Asia GNOME.Asia Steps Load the Pod config ○ Multiple containers Find a node to deploy the pod Create a Pause container Load the CNI config
- 46. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container Load the CNI config from /etc/cni/net.d/…
- 48. COSCUP2018 x openSUSE.Asia GNOME.Asia Steps Load the Pod config ○ Multiple containers Find a node to deploy the pod Create a Pause container Load the CNI config Execute the CNI
- 49. COSCUP2018 x openSUSE.Asia GNOME.Asia Callthebinary Load the binary from the config Find the binary from the /opt/cni/bin/
- 50. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container Call the /opt/cni/bin/flannel Network Connectivity
- 51. COSCUP2018 x openSUSE.Asia GNOME.Asia Steps Load the Pod config ○ Multiple containers Find a node to deploy the pod Create a Pause container Load the CNI config Execute the CNI Create target containers and attach to Pause container
- 52. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container Call the /opt/cni/bin/flannel Network Connectivity Busybox
- 53. COSCUP2018 x openSUSE.Asia GNOME.Asia Attachtoothercontainer docker run –net=… ○ bridge ○ host ○ containerID
- 54. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container Call the /opt/cni/bin/flannel Network Connectivity Busybox Pod
- 56. COSCUP2018 x openSUSE.Asia GNOME.Asia Kubernetesservice There’re three implementation now. ○ User-space Kube-Proxy ○ Kernel-space iptables (default) ○ Kernel-space ipvs We use the iptables to explain how service(ClusterIP) works
- 57. COSCUP2018 x openSUSE.Asia GNOME.Asia Service Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster 10.123.234.56 10.123.234.57 10.123.234.58 App Service Nginx nginx.default endpoints
- 58. COSCUP2018 x openSUSE.Asia GNOME.Asia LAB Get the Service ○ kubectl get service
- 59. COSCUP2018 x openSUSE.Asia GNOME.Asia LAB Get the endpoints ○ kubectl get endpoints
- 60. COSCUP2018 x openSUSE.Asia GNOME.Asia LAB Get the pod ip address ○ kubectl get pods –o wide
- 61. COSCUP2018 x openSUSE.Asia GNOME.Asia Now.Trytofetchthenginx Service name is k8s-nginx-cluster Use the nslookup to lookup the IP ○ nslookup k8s-nginx-cluster ○ nslookup k8s-nginx-cluster.default ■ default is the namespace of the service
- 63. COSCUP2018 x openSUSE.Asia GNOME.Asia Trytofetchthenginx In the pod: curl k8s-nginx-cluster
- 64. COSCUP2018 x openSUSE.Asia GNOME.Asia Howitworks? First, we can get the VIP from the hostname. ○ It’s just a DNS request. Second, we can access the nginx from that VIP address. ○ iptables!!
- 65. COSCUP2018 x openSUSE.Asia GNOME.Asia Lte’swatchtheiptables First, we can use the service name to filter the iptables rules. sudo iptables-save | grep ”k8s- nginx-cluster”
- 66. COSCUP2018 x openSUSE.Asia GNOME.Asia Lte’swatchtheiptables Remember? There’re three endpoints for the service now.
- 67. COSCUP2018 x openSUSE.Asia GNOME.Asia workflowsPackets Packets Packets Match Services’ ClusterIP Find the endpoints DNAT KUBE-SERVICES KUBE-SVC-XXXX KUBE-SEP-XXXX Enter iptables PREROUTING Jump to other chain Jump to other chain Jump to other chain
- 68. COSCUP2018 x openSUSE.Asia GNOME.Asia Howdowechoosewhichonetouse? When we match the clusterIP:Port, goto another custom chain. ○ 10.105.100.214:80
- 69. COSCUP2018 x openSUSE.Asia GNOME.Asia Howdowechoosewhichonetouse? Use the random to choose what endpoint we use.
- 70. COSCUP2018 x openSUSE.Asia GNOME.Asia EP1 EP2 EP3 P < 0.33 P < 0.5 EP1 EP2 EP3 P= 1/3 P= 2/3 * 1/2 = 1/3 P= 2/3 * 1/2 = 1/3
- 71. COSCUP2018 x openSUSE.Asia GNOME.Asia EP1 EP2 EP3 P < 0.2 P < 0.25 EP1 EP2 EP3 EP4 EP5 P < 0.33 P < 0.5 EP4 EP5 P = 0.2 P = 4/5 * 1/4 = 1/5 P = 4/5 * 3/4 * 1/3 = 1/5 P = 4/5 * 3/4 * 2/3 * 1/2 = 1/5 P = 4/5 * 3/4 * 2/3 * 1/2 = 1/5
- 72. COSCUP2018 x openSUSE.Asia GNOME.Asia Howdowechoosewhichonetouse? K8S create a custom chain for each endpoints. First rule is a SNAT ○ The Ngnix want to access outside. Second is DNAT ○ Change the IP to one of the endpoints
- 73. COSCUP2018 x openSUSE.Asia GNOME.Asia Now, We Know The Basic Function Of Kubernetes Network.
- 74. COSCUP2018 x openSUSE.Asia GNOME.Asia What is the next step of kubernetes network
- 75. COSCUP2018 x openSUSE.Asia GNOME.Asia challenges For different use cases ○ 5G/NFV/IoT Network features ○ High performance ○ Low latency Network infrastructure ○ Multiple network ■ Separate the data/control network.
- 76. COSCUP2018 x openSUSE.Asia GNOME.Asia Networkfeaturesc We use the hardware/smart NIC for those requirements before. We also have some mechanism in the software. ○ DPDK ○ SR-IOV ○ … ○ etc
- 77. COSCUP2018 x openSUSE.Asia GNOME.Asia Network Interface Card Linux Kernel Network Stack Network Driver Application Network Interface Card Linux Kernel Network Stack Network Driver Application Kernel Space User Space DPDK
- 78. COSCUP2018 x openSUSE.Asia GNOME.Asia How integrate those with kubernetes?
- 79. COSCUP2018 x openSUSE.Asia GNOME.Asia CNI We use some CNI for those functions. Intel had developed the CNI for those functions. ○ Call sriov-cni ○ https://github.com/intel/sriov-cni
- 80. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2 Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16)
- 81. COSCUP2018 x openSUSE.Asia GNOME.Asia Problem For those container using the DPDK/SR-IOV, it can’t use any kubernetes service now. Since the network function is handled by DPDK/SR-IOV now. How to solve this?
- 82. COSCUP2018 x openSUSE.Asia GNOME.Asia Multus There’s a discussion in the github about that requirement. Intel develop a CNI plugin to support multiple network for a Pod. ○ It’s called Multus CNI Multus call CNIs one by one.
- 83. COSCUP2018 x openSUSE.Asia GNOME.Asia Pod Container eth0 Pod Container eth0 eth1 eth2 Previous What Multus Provide
- 84. COSCUP2018 x openSUSE.Asia GNOME.Asia , , You need to create first
- 85. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2 Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16) Data Network (10.56.10/24)
- 86. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2 Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16) Data Network (10.56.10/24)
- 87. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2 Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16) Data Network (10.56.10/24)
- 88. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2 Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16) Data Network (10.56.10/24)