Kubernetes at Datadog Scale
2 likes364 views
Kubernetes networking can be complex to scale due to issues like growing iptables rules, but newer solutions are helping. Pod networking uses CNI plugins like flannel or Calico to assign each pod an IP and allow communication. Service networking uses kube-proxy and iptables or IPVS for load balancing to pods. DNS is used to resolve service names to IPs. While Kubernetes networking brings flexibility, operators must learn the nuances of their specific CNI plugin and issues can arise, but the ecosystem adapts quickly to new needs and changes don't impact all workloads.
Kubernetes at Datadog Scale
- 1. Kubernetes at Datadog Scale
- 2. Datadog is a monitoring and analytics platform that helps companies improve observability of their infrastructure and applications
- 3. Ara Pulido Developer Advocate at Datadog @arapulido [email protected]
- 5. 19 major releases since 2015 +90k commits +2k contributors
- 6. Kubernetes popularity since 2016
- 7. Kubernetes popularity since 2016
- 8. Kubernetes vs Mariah Carey
- 9. My journey with Kubernetes 2 YEARS FULL TIME IN KUBERNETES PROJECTS MAINTAINER OF A YOUTUBE KUBERNETES CHANNEL CKA; PART OF THE TEAM THAT CREATED CKAD
- 12. Dozens of clusters +1K nodes / cluster Multicloud
- 13. Survey: number of nodes / cluster Source: https://github.com/MagalixCorp/kubernetes-autoscale-survey
- 14. What are the common scalability issues?
- 15. Networking
- 17. Kubernetes Networking 101 POD NETWORKING DNS SERVICE NETWORKING
- 18. Pod Networking Pod 172.17.0.2 Pod 172.17.0.3 EVERY POD GETS A UNIQUE IP EVERY POD CAN TALK TO ANY OTHER POD IN THE CLUSTER
- 19. Service Networking Client 172.17.0.6 Pod 172.17.0.4 Pod 172.17.0.5 Service 10.107.122.52 kube-proxy
- 21. Pod Networking
- 22. Pod Networking Pod 172.17.0.2 Pod 172.17.0.3 EVERY POD GETS A UNIQUE IP EVERY POD CAN TALK TO ANY OTHER POD IN THE CLUSTER
- 24. Overlays (flannel, weavenet) Pod 10.244.2.1 10.244.1.0/24 10.244.0.0/16 10.244.2.0/24 10.244.0.0/16 Pod 10.244.1.1 192.168.0.1 192.168.0.2 cni0 flannel0 cni0 flannel0
- 25. Overlays (flannel, weavenet) Pod 10.244.2.1 10.244.1.0/24 10.244.0.0/16 10.244.2.0/24 10.244.0.0/16 Pod 10.244.1.1 192.168.0.1 192.168.0.2 cni0 flannel0 cni0 flannel0 I WAS HERE
- 26. Overlays (flannel, weavenet) Pod 10.244.2.1 Pod 10.244.1.1 192.168.0.1 192.168.0.2 10.244.1.0/24 10.244.0.0/16 10.244.2.0/24 10.244.0.0/16 cni0 flannel0 cni0 flannel0
- 27. Overlays (flannel, weavenet) Pod 10.244.2.1 Pod 10.244.1.1 192.168.0.1 192.168.0.2 10.244.1.0/24 10.244.0.0/16 10.244.2.0/24 10.244.0.0/16 cni0 flannel0 cni0 flannel0
- 29. Overlays 10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.0 10.244.0.0/16 dev flannel proto kernel scope link src 10.244.1.1 Pod 10.244.2.1 10.244.1.0/24 10.244.0.0/16 10.244.2.0/24 10.244.0.0/16 cni0 flannel0 cni0 flannel0 Pod 10.244.1.1 192.168.0.1 192.168.0.2 Src: 10.244.1.1 Dst: 10.244.2.1
- 30. Overlays Src: 192.168.0.1 Dst: 192.168.0.2 Src: 10.244.1.1 Dst: 10.244.2.1 Pod 10.244.2.1 10.244.1.0/24 10.244.0.0/16 10.244.2.0/24 10.244.0.0/16 cni0 flannel0 cni0 flannel0 Pod 10.244.1.1 192.168.0.1 192.168.0.2 10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.0 10.244.0.0/16 dev flannel proto kernel scope link src 10.244.1.1
- 31. BGP (Calico) Pod 10.244.162.131/32 10.244.143.129/32 Pod 10.244.162.131 192.168.0.1 192.168.0.2 BGP BGP 10.244.143.129 calixxxx calixxxx
- 32. BGP (Calico) 10.244.162.131 dev cali12d4a061371 scope link 10.244.143.129 via 192.168.0.2 dev eth0 10.244.143.129 dev cali12d4bf881 scope link 10.244.162.131 via 192.168.0.1 dev eth0 Pod 10.244.162.131/32 10.244.143.129/32calixxxx calixxxx Pod 10.244.162.131 192.168.0.1 192.168.0.2 BGP BGP 10.244.143.129
- 33. BGP (Calico) 10.244.162.131 dev cali12d4a061371 scope link Pod 10.244.162.131/32 10.244.143.129/32calixxxx calixxxx Pod 10.244.162.131 192.168.0.1 192.168.0.2 BGP BGP 10.244.143.129
- 34. BGP (Calico) 10.244.143.129 via 192.168.0.2 dev eth0 Pod 10.244.162.131/32 10.244.143.129/32calixxxx calixxxx Pod 10.244.162.131 192.168.0.1 192.168.0.2 BGP BGP 10.244.143.129
- 35. Direct Pod Routing (AWS, Lyft) Pod 10.0.0.3 Pod 10.0.0.4 SDN
- 36. Direct Pod Routing (AWS, Lyft) Pod 10.0.0.3 Pod 10.0.0.4 SDN DATADOG
- 37. Direct Pod Routing (AWS, Lyft) Pod 10.0.0.3 Pod 10.0.0.4 SDN
- 38. Be ready to become an “expert” in the CNI plugin of your choice
- 40. Client 172.17.0.6 Pod 172.17.0.4 Pod 172.17.0.5 Service 10.107.122.52 kube-proxy Service Networking
- 41. iptables mode (default) Client 172.17.0.6 Pod 1 172.17.0.4 Pod 2 172.17.0.5 Iptables 10.107.122.52 kube-proxy API Watch svc, endpoints
- 42. iptables mode (default) Client 172.17.0.6 Pod 1 172.17.0.4 Pod 2 172.17.0.5 Iptables 10.107.122.52 kube-proxy API Watch svc, endpoints I WAS HERE
- 43. iptables mode (default) Client 172.17.0.6 Pod 1 172.17.0.4 Pod 2 172.17.0.5 Iptables 10.107.122.52 kube-proxy API Watch svc, endpoints
- 44. iptables mode (default) Chain KUBE-SERVICES (2 references) KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- anywhere 10.107.122.52 /* default/nginx: cluster IP */ tcp dpt:www Chain KUBE-SVC-4N57TFCL4MD7ZTDA (1 references) target prot opt source destination KUBE-SEP-UZXILYFQQ2IZUWN5 all -- anywhere anywhere statistic mode random probability 0.50000000000 KUBE-SEP-43IWXJI557JKCKCF all -- anywhere anywhere Chain KUBE-SEP-43IWXJI557JKCKCF (1 references) target prot opt source destination KUBE-MARK-MASQ all -- 172.17.0.5 anywhere DNAT tcp -- anywhere anywhere tcp to:172.17.0.5:80
- 45. iptables mode (default) Chain KUBE-SERVICES (2 references) KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- anywhere 10.107.122.52 /* default/nginx: cluster IP */ tcp dpt:www
- 46. iptables mode (default) Chain KUBE-SERVICES (2 references) KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- anywhere 10.107.122.52 /* default/nginx: cluster IP */ tcp dpt:www Chain KUBE-SVC-4N57TFCL4MD7ZTDA (1 references) target prot opt source destination KUBE-SEP-UZXILYFQQ2IZUWN5 all -- anywhere anywhere statistic mode random probability 0.50000000000 KUBE-SEP-43IWXJI557JKCKCF all -- anywhere anywhere
- 47. iptables mode (default) Chain KUBE-SERVICES (2 references) KUBE-SVC-4N57TFCL4MD7ZTDA tcp -- anywhere 10.107.122.52 /* default/nginx: cluster IP */ tcp dpt:www Chain KUBE-SVC-4N57TFCL4MD7ZTDA (1 references) target prot opt source destination KUBE-SEP-UZXILYFQQ2IZUWN5 all -- anywhere anywhere statistic mode random probability 0.50000000000 KUBE-SEP-43IWXJI557JKCKCF all -- anywhere anywhere Chain KUBE-SEP-43IWXJI557JKCKCF (1 references) target prot opt source destination KUBE-MARK-MASQ all -- 172.17.0.5 anywhere DNAT tcp -- anywhere anywhere tcp to:172.17.0.5:80
- 48. GROWS LINEARLY WITH PODS AND SERVICES THERE IS NO ROOM FOR MORE FEATURES FOR EVERY CHANGE, THE WHOLE TABLE NEEDS TO BE RESYNC iptables mode (default)
- 49. GA SINCE 1.11 IPVS mode
- 50. GA SINCE 1.11 IPVS mode DATADOG
- 51. ATOMIC CHANGES (NO NEED TO RESYNC EVERY RULE) O(1) COMPLEXITY DIFFERENT LOAD BALANCING ALGORITHMS ipvsadm -ln --tcp-service 10.107.122.52:80 Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.107.122.52:80 rr -> 172.17.0.4:80 Masq 1 0 37 -> 172.17.0.5:80 Masq 1 0 44 GA SINCE 1.11 IPVS mode
- 52. GA SINCE 1.11 IPVS mode
- 53. Many large deployments are already moving to IPVS and newer solutions are being developed (eBPF based)
- 54. DNS
- 57. NodeLocal DNSCache is now part of Kubernetes (Beta 1.15)
- 58. DNS Local Cache ` Client DNS Cache CoreDNS DATADOG
- 59. Take aways
- 60. The bad
- 61. KUBERNETES IS FLEXIBLE FOR DEVS, COMPLEX FOR OPS YOU WILL HIT BUGS CNI PLUGINS WORK *VERY* DIFFERENTLY. LEARN YOURS.
- 62. The good
- 63. THE ECOSYSTEM PICKS UP QUICKLY DEVELOPMENT EXPERIENCE DOESN’T CHANGE (MUCH) CHANGES DON’T HAVE TO HAPPEN FOR ALL WORKLOADS DIRECTLY
- 64. Thank you!