KubeCon EU 2016: Creating an Advanced Load Balancing Solution for Kubernetes with NGINX
14 likes236,038 views
The document outlines the creation of an advanced load balancing solution for Kubernetes using NGINX, covering basic and advanced load balancing techniques and various service types like NodePort, LoadBalancer, and Ingress. Key features highlighted include SSL termination, health checks, session persistence, and content-based routing, along with examples of configurations and commands. Additionally, it emphasizes the deployment of an Ingress controller for achieving improved traffic management and scaling in Kubernetes environments.
KubeCon EU 2016: Creating an Advanced Load Balancing Solution for Kubernetes with NGINX
- 1. Creating an Advanced Load Balancing Solution for Kubernetes with NGINX Andrew Hutchings — Technical Product Manager, NGINX, Inc., @LinuxJedi
- 2. About LinuxJedi • Kubernetes user for 4 days • Worked at HP on OpenStack LBaaS and ATG • Worked on several Open Source DBs • Alopecia sufferer
- 3. Goals • Basic and advanced load balancing • Current load balancing options in Kubernetes • Ingress resource • Implementing an Ingress controller for NGINX • Load balancing demo: exposing Kubernetes services to the Internet
- 4. Basic Load Balancing A load balancer distributes request among healthy servers LB Server 1 Server 2 Server 3
- 5. Basic Load Balancing HTTPHTTP Layer 7 TCPTCP UDPUDP Layer 4
- 6. Advanced Load Balancing • SSL termination • Active health checks • Security • Bandwidth limits • Logging • Real-time statistics • Session Persistence • Content-based routing • and more…
- 7. Load Balancing in Kubernetes Internal • kube-proxy External • NodePort • LoadBalancer • External IPs • Service loadbalancer • Ingress
- 8. Internal: Kube-proxy apiVersion: v1 kind: Service metadata: name: backend-service spec: ports: - port: 80 targetPort: 80 protocol: TCP selector: app: backend apiVersion: v1 kind: Service metadata: name: backend-service spec: ports: - port: 80 targetPort: 80 protocol: TCP selector: app: backend # env | grep -i backend BACKEND_SERVICE_SERVICE_HOST=10.3.246.245 BACKEND_SERVICE_SERVICE_PORT=80 … # env | grep -i backend BACKEND_SERVICE_SERVICE_HOST=10.3.246.245 BACKEND_SERVICE_SERVICE_PORT=80 … # nslookup backend-service … Name: backend-service Address 1: 10.3.246.245 # nslookup backend-service … Name: backend-service Address 1: 10.3.246.245 $ kubectl get svc NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE backend-service 10.3.246.245 <none> 80/TCP app=backend 6m $ kubectl get svc NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE backend-service 10.3.246.245 <none> 80/TCP app=backend 6m
- 9. Internal: Kube-proxy kube-proxykube-proxy BB kube-proxykube-proxy BB kube-proxykube-proxy BB Features • TCP/UDP • Health checks • Client IP session affinity
- 10. External: NodePort apiVersion: v1 kind: Service metadata: name: backend-service spec: type: NodePort ports: - port: 80 targetPort: 80 protocol: TCP selector: app: backend apiVersion: v1 kind: Service metadata: name: backend-service spec: type: NodePort ports: - port: 80 targetPort: 80 protocol: TCP selector: app: backend $ kubectl create -f backend-service-nodeport.yaml You have exposed your service on an external port on all nodes in your cluster. If you want to expose this service to the external internet, you may need to set up firewall rules for the service port(s) (tcp:31107) to serve traffic. $ kubectl create -f backend-service-nodeport.yaml You have exposed your service on an external port on all nodes in your cluster. If you want to expose this service to the external internet, you may need to set up firewall rules for the service port(s) (tcp:31107) to serve traffic.
- 11. External: NodePort Features • TCP/UDP • Health checks kube-proxykube-proxykube-proxykube-proxy BB kube-proxykube-proxy BB NodePortNodePort NodePortNodePort NodePortNodePort BB
- 12. External: LoadBalancer apiVersion: v1 kind: Service metadata: name: backend-service spec: type: LoadBalancer ports: - port: 80 targetPort: 80 protocol: TCP selector: app: backend apiVersion: v1 kind: Service metadata: name: backend-service spec: type: LoadBalancer ports: - port: 80 targetPort: 80 protocol: TCP selector: app: backend $ kubectl describe svc backend-service Name: backend-service Namespace: default Labels: <none> Selector: app=backend Type: LoadBalancer IP: 10.3.249.155 LoadBalancer Ingress: XXX.YYY.ZZZ.III Port: <unnamed> 80/TCP NodePort: <unnamed> 32074/TCP Endpoints: <none> Session Affinity: None $ kubectl describe svc backend-service Name: backend-service Namespace: default Labels: <none> Selector: app=backend Type: LoadBalancer IP: 10.3.249.155 LoadBalancer Ingress: XXX.YYY.ZZZ.III Port: <unnamed> 80/TCP NodePort: <unnamed> 32074/TCP Endpoints: <none> Session Affinity: None
- 13. External: LoadBalancer Features • TCP • Health checks • Client IP session affinity (GCE) kube-proxykube-proxykube-proxykube-proxy BB kube-proxykube-proxy BB NodePortNodePort NodePortNodePort NodePortNodePort BB Cloud LB Cloud LB
- 14. External: External IPs apiVersion: v1 kind: Service metadata: name: backend-service spec: externalIPs: - 10.240.0.2 - 10.240.0.3 ports: - port: 80 targetPort: 80 protocol: TCP selector: app: backend apiVersion: v1 kind: Service metadata: name: backend-service spec: externalIPs: - 10.240.0.2 - 10.240.0.3 ports: - port: 80 targetPort: 80 protocol: TCP selector: app: backend $ kubectl get nodes -o json | grep -A 1 "InternalIP" "type": "InternalIP", "address": "10.240.0.2" -- "type": "InternalIP", "address": "10.240.0.3" -- "type": "InternalIP", "address": "10.240.0.4" $ kubectl get nodes -o json | grep -A 1 "InternalIP" "type": "InternalIP", "address": "10.240.0.2" -- "type": "InternalIP", "address": "10.240.0.3" -- "type": "InternalIP", "address": "10.240.0.4"
- 15. External: External IPs Features • TCP/UDP • Health checks kube-proxykube-proxykube-proxykube-proxy BB kube-proxykube-proxy BB BB 8080 10.240.0.2 10.240.0.3 10.240.0.4 8080
- 16. External: service LoadBalancer https://github.com/kubernetes/contrib/tree/master/service-loadbalancer 1 or more HAProxy, each deployed in a pod Services -> HAProxy configuration svcA-> /svcA svcB -> /svcB Features • TCP/UDP, HTTP • URL Mapping • SSL Termination (via Annotations) • Session Persistence (via Annotations) • Multiple algorithms (via Annotations)
- 17. External: Ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name: hello-ingress spec: rules: - host: hello.example.com http: paths: - path: /a backend: serviceName: backend-a servicePort: 80 - path: /b backend: serviceName: backend-b servicePort: 8080 apiVersion: extensions/v1beta1 kind: Ingress metadata: name: hello-ingress spec: rules: - host: hello.example.com http: paths: - path: /a backend: serviceName: backend-a servicePort: 80 - path: /b backend: serviceName: backend-b servicePort: 8080 • hello.example/a -> backend-a:80 • hello.example/b -> backend-b:8080
- 18. External: Ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name: hello-ingress spec: tls: - hosts: - hello.example.com secretName: hello-secret rules: - host: hello.example.com . . . apiVersion: extensions/v1beta1 kind: Ingress metadata: name: hello-ingress spec: tls: - hosts: - hello.example.com secretName: hello-secret rules: - host: hello.example.com . . . apiVersion: v1 kind: Secret metadata: name: hello-secret type: Opaque data: tls.crt: <base-64 encoded crt> tls.key: <base-64 encoded key> apiVersion: v1 kind: Secret metadata: name: hello-secret type: Opaque data: tls.crt: <base-64 encoded crt> tls.key: <base-64 encoded key> New in 1.2: TLS support
- 19. External: Ingress Features • HTTP Load Balancing • SSL Termination • Content-based routing How to use it Ingress Controller must be deployed
- 20. External: Ingress Ingress Controller Ingress Controller Ingress Resources Ingress Resources Load BalancerLoad Balancer watches configures
- 21. External: Ingress Cloud Load Balancers • GCE HTTP Load Balancer Software Load Balancers • NGINX https://github.com/kubernetes/contrib/tree/master/ingress/controllers
- 22. NGINX • Layer 4/Layer 7 Load Balancer • Advanced algorithms • SSL termination • Content-based routing • Limits • HTTP/2 gateway • Logging • Security • Real-time statistics* • Layer 7 Session Persistence* • Dynamic reconfiguration* * NGINX Plus Also a webserver and cache
- 23. NGINX Ingress Controller apiVersion: extensions/v1beta1 kind: Ingress metadata: name: hello-ingress spec: rules: - host: hello.example.com http: paths: - path: /a backend: serviceName: backend-a servicePort: 80 - path: /b backend: serviceName: backend-b servicePort: 8080 apiVersion: extensions/v1beta1 kind: Ingress metadata: name: hello-ingress spec: rules: - host: hello.example.com http: paths: - path: /a backend: serviceName: backend-a servicePort: 80 - path: /b backend: serviceName: backend-b servicePort: 8080 upstream backend-a { server 10.3.246.245:80; } upstream backend-b { server 10.3.246.249:8080; } server { listen 80; server_name hello.example.com; location /a { proxy_pass http://backend-a; } location /b { proxy_pass http://backend-b; } } upstream backend-a { server 10.3.246.245:80; } upstream backend-b { server 10.3.246.249:8080; } server { listen 80; server_name hello.example.com; location /a { proxy_pass http://backend-a; } location /b { proxy_pass http://backend-b; } }
- 24. NGINX Ingress Controller 1. Watch for Ingress resources 2. Watch for Services and Endpoints: to get IP address of a service or its endpoints in case of a headless service 3. Watch for Secrets
- 25. NGINX Ingress Controller IngressIngress EndpointsEndpoints ServiceService SecretSecret IngressIngress affects Changes in 1. Regenerate configuration for the Ingress 2. Reload NGINX
- 26. NGINX Ingress Controller • NGINX Plus supports re-resolving DNS names in runtime every X seconds • Doesn’t fail when a name can’t be resolved • Simplifies implementation: no need to watch for Services and Endpoints
- 27. NGINX Ingress Controller • As an example we took the GCE HTTP Load Balancer Ingress Controller — https://github.com/kubernetes/contrib/tree/master/ingress/controllers/ gce • Written in Go • Different implementations for NGINX and NGINX Plus • Deployed in the same container as NGINX. the Controller starts first and then launches NGINX.
- 28. NGINX Ingress Controller • HTTP Load Balancing • SSL Termination • Content-based routing Features • Advanced algorithms • Limits • Access Control • Logging • Limits Real-time statistics (NGINX Plus) • Layer 7 Session Persistence (NGINX Plus) • Dynamic reconfiguration (NGINX Plus) • and more Features, supported by changing NGINX templates
- 30. Demo • tea-rc and tea-svc • coffee-rc and headless coffee-svc • Ingress resource cafe-ingress with TLS • Secret cafe-secret • NGINX Plus Ingress Controller nginx-plus-ingress-rc
- 31. NGINX Ingress Controller • Expose more NGINX features via Kubernetes resources (Annotations and Config Maps) • Make it production-ready • Improve it based on your feedback Wishlist
- 32. The End ● Resources: http://tiny.cc/nginx-ingress ● NGINX: https://www.nginx.com/ ● My site: http://linuxjedi.co.uk/ ● Twitter: @LinuxJedi ● Freenode: LinuxJedi ● Email: [email protected]