Kubernetes Networking
- 1. Kubernetes Networking Seattle Kubernetes Meetup CJ Cullen <[email protected]> Software Engineer @cj_cullen github.com/cjcullen
- 13. Host ports A: 172.16.1.1 3306 B: 172.16.1.2 80 9376 11878SNAT SNAT C: 172.16.1.1 8000
- 14. Host ports A: 172.16.1.1 3306 B: 172.16.1.2 80 9376 11878SNAT SNAT C: 172.16.1.1 8000 REJECTED
- 16. Kubernetes networking IPs are routable • vs docker default private IP Pods can reach each other without NAT • even across nodes No brokering of port numbers • too complex, why bother? This is a fundamental requirement • can be L3 routed • can be underlayed (cloud) • can be overlayed (SDN)
- 19. Kubernetes networking On GCE/GKE • GCE Advanced Routes (program the fabric) • “Everything to 10.1.1.0/24, send to this VM” Plenty of other ways • AWS: Route Tables • Weave • Calico • Flannel • OVS • OpenContrail • Cisco Contiv • Others...
- 20. Kubernetes networking On GCE/GKE • GCE Advanced Routes (program the fabric) • “Everything to 10.1.1.0/24, send to this VM” Plenty of other ways • AWS: Route Tables • Weave • Calico • Flannel • OVS • OpenContrail • Cisco Contiv • Others...
- 21. Kubernetes networking On GCE/GKE • GCE Advanced Routes (program the fabric) • “Everything to 10.1.1.0/24, send to this VM” Plenty of other ways • AWS: Route Tables • Weave • Calico • Flannel • OVS • OpenContrail • Cisco Contiv • Others...
- 22. Pods
- 23. Pods Small group of containers & volumes Tightly coupled The atom of scheduling & placement Shared namespace • share IP address & localhost • share IPC, etc. Managed lifecycle • bound to a node, restart in place • can die, cannot be reborn with same ID Example: data puller & web server Consumers Content Manager File Puller Web Server Volume Pod
- 24. Pods Small group of containers & volumes Tightly coupled The atom of scheduling & placement Shared namespace • share IP address & localhost • share IPC, etc. Managed lifecycle • bound to a node, restart in place • can die, cannot be reborn with same ID Example: data puller & web server 10.1.1.2
- 25. Pods Small group of containers & volumes Tightly coupled The atom of scheduling & placement Shared namespace • share IP address & localhost • share IPC, etc. Managed lifecycle • bound to a node, restart in place • can die, cannot be reborn with same ID Example: data puller & web server c1 --net=container:infra --ipc=container:infra infra 10.1.1.2 c2 --net=container:infra --ipc=container:infra
- 26. Services
- 27. Services A group of pods that work together • grouped by a selector Defines access policy • “load balanced” or “headless” Gets a stable virtual IP and port • sometimes called the service portal • also a DNS name VIP is managed by kube-proxy • watches all services • updates iptables when backends change Hides complexity - ideal for non-native apps Client Virtual IP
- 28. kube-proxy
- 30. iptables apiserver Node X watch services & endpoints kube-proxy (legacy) kube-proxy
- 31. iptables apiserver Node X kubectl run ... watch kube-proxy (legacy) kube-proxy
- 33. iptables apiserver Node X watch kubectl expose ... kube-proxy (legacy) kube-proxy
- 45. kube-proxy (legacy) Userspace proxy isn’t ideal Burns CPU copying bytes • “Proxy” is just parallel copy loops. Loses source IP • Everything looks like it’s from the node IP. Userspace TCP listening = higher latency
- 48. iptables kube-proxy iptables kube-proxy apiserver Node X watch services & endpoints
- 49. iptables kube-proxy iptables kube-proxy apiserver Node X kubectl run ... watch
- 50. iptables kube-proxy iptables kube-proxy apiserver Node X schedule watch
- 51. iptables kube-proxy iptables kube-proxy apiserver Node X watch kubectl expose ...
- 52. iptables kube-proxy iptables kube-proxy apiserver Node X new service! update
- 53. iptables kube-proxy iptables kube-proxy apiserver Node X watch configure
- 54. iptables kube-proxy iptables kube-proxy apiserver Node X watch VIP
- 55. iptables kube-proxy iptables kube-proxy apiserver Node X new endpoints! update VIP
- 56. iptables kube-proxy iptables kube-proxy apiserver Node X VIP watch configure
- 57. iptables kube-proxy iptables kube-proxy apiserver Node X VIP watch
- 58. iptables kube-proxy iptables kube-proxy apiserver Node X VIP watch Client
- 59. iptables kube-proxy iptables kube-proxy apiserver Node X VIP watch Client
- 60. iptables kube-proxy iptables kube-proxy apiserver Node X VIP watch Client
- 61. iptables kube-proxy iptables kube-proxy apiserver Node X VIP watch Client
- 62. iptables kube-proxy Mean Latency contrib/for-tests/netperf-tester --number=1000 Mean Latency Microseconds iptables kube-proxy legacy kube-proxy
- 63. Services are just an abstraction • Only requirement: route (and maybe load balance) a virtual IP to a set of backends. Kube-proxy is an implementation • Kube-proxy watches apiserver. • iptables is re-configured on changes. There could be other ways • Userspace, iptables, IP Virtual Servers? Services
- 64. DNS Run SkyDNS as a pod in the cluster • kube2sky bridges Kubernetes API -> SkyDNS • Tell kubelets about it (static service IP) Strictly optional, but practically required • LOTS of things depend on it • Probably will become more integrated Or plug in your own! kubernetes kubernetes.default kubernetes.default.svc.cluster.local foo.my-namespace.svc.cluster.local
- 65. DNS Run SkyDNS as a pod in the cluster • kube2sky bridges Kubernetes API -> SkyDNS • Tell kubelets about it (static service IP) Strictly optional, but practically required • LOTS of things depend on it • Probably will become more integrated Or plug in your own! apiserverwatch etcd kube-dns-qxin kube2skyskyDNS
- 66. DNS Run SkyDNS as a pod in the cluster • kube2sky bridges Kubernetes API -> SkyDNS • Tell kubelets about it (static service IP) Strictly optional, but practically required • LOTS of things depend on it • Probably will become more integrated Or plug in your own! nameserver 10.0.0.10 ... /etc/resolv.conf apiserverwatch etcd kube-dns-qxin kube2skyskyDNS
- 67. DNS Run SkyDNS as a pod in the cluster • kube2sky bridges Kubernetes API -> SkyDNS • Tell kubelets about it (static service IP) Strictly optional, but practically required • LOTS of things depend on it • Probably will become more integrated Or plug in your own! nameserver 10.0.0.10 ... /etc/resolv.conf apiserverwatch etcd kube-dns-qxin kube2skyskyDNS 10.0.0.10
- 68. Putting it Together What happens when I... $ curl foo.my-namespace Client
- 69. What happens when I... $ curl foo.my-namespace Putting it Together nameserver 10.0.0.10 ... /etc/resolv.conf Client 10.1.0.1
- 70. 10.1.0.1 Putting it Together What happens when I... $ curl foo.my-namespace etcd kube-dns-qxin kube2skyskyDNS 10.0.0.10 foo.my-namespace? Client
- 71. Putting it Together What happens when I... $ curl foo.my-namespace etcd kube-dns-qxin kube2skyskyDNS 10.0.0.10 10.0.123.45 Client 10.1.0.1
- 72. Putting it Together What happens when I... $ curl foo.my-namespace 10.0.123.45Client 10.1.0.1
- 73. Putting it Together What happens when I... $ curl foo.my-namespace Client VIP10.0.123.45 10.1.0.1
- 74. Putting it Together What happens when I... $ curl foo.my-namespace Client VIP10.0.123.45 iptables 10.1.0.1
- 75. Putting it Together What happens when I... $ curl foo.my-namespace Client VIP10.0.123.45 iptables 10.1.0.1 10.1.0.6 10.1.3.1 10.1.6.3
- 76. Putting it Together What happens when I... $ curl foo.my-namespace Client VIP10.0.123.45 iptables 10.1.3.1 10.1.0.1 10.1.0.6 10.1.3.1 10.1.6.3
- 77. Putting it Together What happens when I... $ curl foo.my-namespace Client VIP10.0.123.45 iptables 10.1.3.1 10.1.0.1 10.1.0.6 10.1.3.1 10.1.6.3 10.1.3.0/24 -> Node X
- 78. Putting it Together What happens when I... $ curl foo.my-namespace Client VIP10.0.123.45 iptables 10.1.3.1 10.1.0.1 10.1.0.6 10.1.3.1 10.1.6.3
- 79. Putting it Together What happens when I... $ curl foo.my-namespace Client VIP10.0.123.45 iptables 10.1.3.1 10.1.0.1 10.1.0.6 10.1.3.1 10.1.6.3 Hello World!
- 80. Putting it Together What happens when I... $ curl foo.my-namespace Client iptables Hello World! 10.1.0.1 10.1.0.6 10.1.3.1 10.1.6.3 10.1.0.1
- 81. Putting it Together What happens when I... $ curl foo.my-namespace Client iptables Hello World! 10.1.0.1 10.1.0.6 10.1.3.1 10.1.6.3 10.1.0.0/24 -> Node Y 10.1.0.1
- 82. Putting it Together What happens when I... $ curl foo.my-namespace Client iptables Hello World! 10.1.0.1 10.1.0.6 10.1.3.1 10.1.6.3 10.1.0.0/24 -> Node Y 10.1.0.1
- 83. Putting it Together What happens when I... $ curl foo.my-namespace Hello World! Client iptables Hello World! 10.1.0.1 10.1.0.6 10.1.3.1 10.1.6.3 10.1.0.0/24 -> Node Y 10.1.0.1
- 85. External Services Services IPs are only available inside the cluster Need to receive traffic from “the outside world” Builtin: Service “type” • nodePort: expose on a port on every node • loadBalancer: provision a cloud load-balancer DiY load-balancer solutions • socat (for nodePort remapping) • haproxy • nginx
- 87. Ingress (L7) Services are assumed L3/L4 Lots of apps want HTTP/HTTPS Ingress maps incoming traffic to backend services • by HTTP host headers • by HTTP URL paths HAProxy and GCE implementations No SSL yet Status: BETA in Kubernetes v1.1 URL Map Client
- 88. Ingress (L7) Services are assumed L3/L4 Lots of apps want HTTP/HTTPS Ingress maps incoming traffic to backend services • by HTTP host headers • by HTTP URL paths HAProxy and GCE implementations No SSL yet Status: BETA in Kubernetes v1.1 URL Map Client api.company.com api.company.com/foo api.company.com/bar othercompany.com/*
- 89. Network Plugins
- 90. Network Plugins Introduced in Kubernetes v1.0 • VERY experimental Uses CNI (CoreOS) in v1.1 • Simple exec interface • Not using Docker libnetwork • but can defer to Docker for networking Cluster admins can customize their installs • DHCP, MACVLAN, Flannel, custom net Plugin Plugin Plugin
- 91. Kubernetes is Open - open community - open design - open source - open to ideas Networking is Hard - help guide us! http://kubernetes.io https://github.com/kubernetes/kubernetes slack: kubernetes twitter: @kubernetesio