Kubernetes Architecture and Introduction

#k8sber @the1stein @mieszkoman © 2015 Mesosphere, Inc. All Rights Reserved. 1
Kubernetes
Architecture & Introduction
Dr. Stefan Schimanski, Sergiusz Urbaniak
hash tag #k8sber
follow @kubernetesBER

Engineers @ Mesosphere
Working on Kubernetes-on-Mesos
https://github.com/kubernetes/kubernetes/tree/master/contrib/mesos
Who are we?
Stefan
@the1stein
sttts@github
Sergiusz
@mieszkoman
s-urbaniak@github

The company behind:
● Apache Mesos (https://mesos.apache.org/)
A distributed systems kernel.
● Marathon (https://mesosphere.github.io/marathon/)
An Apache Mesos framework
for long-running applications.
● Chronos (http://mesos.github.io/chronos/)
A distributed cron replacement.
Who/What is Mesosphere?
DCOS
(https://mesosphere.com/)
A data center operating system.

- Archeology: before and without Kubernetes
- Deployment: kube-up, DCOS, GKE
- Core Architecture: the apiserver, the kubelet and the scheduler
- Compute Model: the pod, the service and the controller
No whitepaper talk. We dive deep!
Next 45 Minutes

Python based agent to deploy declarative pods using Docker.
How it all started
googlearchive / container-agent
...
Kubernetes:

[...] container group defined by the manifest to share:
● Network Namespaces
● Volumes
This creates a runtime environment where:
● Containers can connect to a service running in other containers of the same group
using localhost and a fixed port.
● Containers of the same group can't run services on the same ports.
● Containers of the same group can mount shared volumes defined in the manifest.
A Pod was described as a ...
googlearchive / container-agent

[...] container group defined by the manifest to share:
● Network Namespaces
● Volumes
This creates a runtime environment where:
● Containers can connect to a service running in other containers of the same group
using localhost and a fixed port.
● Containers of the same group can't run services on the same ports.
● Containers of the same group can mount shared volumes defined in the manifest.
A Pod was described as a ...
googlearchive / container-agent Still true with
Kubernetes

$ docker run -it --name mypod busybox sleep 9999999
$ docker run -it --ipc=container:mypod --net=container:mypod
-v /tmp/mypod:/tmp/mypod
-w /tmp/mypod ubuntu python3 -m http.server 80
$ docker run -it --ipc=container:mypod --net=container:mypod
-v /tmp/mypod:/tmp/mypod
sttts/busybox-curl /bin/sh
A Pod in Docker terms:
Back to the roots
“root”, “pause” or
“infrastructure” container
“webserver”
“busybox”, our worker

A Pod in Docker terms – visually
● Note: different OSes, resource constraints, restart policies
mypod
eth0: 192.168.66.6
lo: 127.0.0.1
ipc
net
/tmp/mypod
webserver:80
busybox
sleep

while true; do
curl -k 'https://mobile.twitter.com/search?q=%23kubernetes%20%23berlin&s=typd'
> /tmp/mypod/index.html
sleep 10
done
A little Twitter Mirror Pod
mypod
eth0: 192.168.66.6
lo: 127.0.0.1
ipc
net
/tmp/mypod
webserver:80
busybox
sleep
http 192.168.66.6:80

$ git clone git@github.com:kubernetes/kubernetes.git
$ cd kubernetes
$ build/run.sh hack/build-go.sh && make
$ KUBERNETES_PROVIDER=mesos/docker cluster/kube-up.sh
$ alias kubectl=_output/local/bin/darwin/amd64/kubectl
$ kubectl get pods
Get a Kubernetes cluster: mesos/docker with kube-up

$ dcos config prepend package.sources
https://github.com/mesosphere/multiverse/archive/version-1.x.zip
$ dcos package install kubernetes
$ dcos kubectl create -f nginx.yml
pods/nginx
$ dcos kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 1m
Package for
Mesosphere’s
DCOS

Google
CKontainer
Engine
● install the gcloud cli
● prepare your Google Cloud account with a project with
enabled GCE APIs, e.g. “kube-test-1154”
$ gcloud components install kubectl
$ gcloud container clusters create demo-cluster --num-nodes=2
$ gcloud config set container/cluster demo-cluster
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 1m

Getting Started Guides

$ git clone git@github.com:kubernetes/kubernetes.git
$ cd kubernetes
$ build/run.sh hack/build-go.sh && make
$ KUBERNETES_PROVIDER=mesos/docker cluster/kube-up.sh
$ alias kubectl=_output/local/bin/darwin/amd64/kubectl
$ kubectl get pods
Get a Kubernetes cluster: mesos/docker with kube-up

Kubernetes Core Architecture – the apiserver
apiserver
default namespace
kube-system namespace
kube-dns fluentdkube-ui
PodsPodsPods
PodsPodsServices
PodsPodsRCs
PodsPodsSecrets
PodsPodsVolumesPodsPodsJobs
...
...
...
etcd clu ster
store
state
● all the cluster state
● no logic, only storage and API

Kubernetes Core Architecture – the apiserver
apiserver
default namespace
PodsPodsPods
PodsPodsServices
PodsPodsRCs
PodsPodsSecrets
...
...
...RESTAPI
kubectl
HTTP Kubernetes
cluster
etcd clu ster
store
state

#k8sber @the1stein @mieszkoman © 2015 Mesosphere, Inc. All Rights Reserved.
node Anode A
18
Kubernetes Core Architecture – the components
kubectl
HTTP
apiserver
default namespace
PodsPodsPods
PodsPodsServices
PodsPodsRCs
PodsPodsSecrets
...
...
...
watches
for new
pods
node A kubelet
Scheduler
assigns pods
to nodes
etcd clu ster
store
state
launches pods
with Docker or rkt
Controller
Manager

kubectl - interacting with a Kubernetes cluster
Usage:
kubectl [flags]
kubectl [command]
Available Commands:
get Display one or many resources
describe Show details of a specific resource or group of resources
create Create a resource by filename or stdin
replace Replace a resource by filename or stdin.
patch Update field(s) of a resource by stdin.
delete Delete resources by filenames, stdin, resources and names, or by resources and label selector.
edit Edit a resource on the server
apply Apply a configuration to a resource by filename or stdin
namespace SUPERSEDED: Set and view the current Kubernetes namespace
logs Print the logs for a container in a pod.
rolling-update Perform a rolling update of the given ReplicationController.
scale Set a new size for a Replication Controller.
attach Attach to a running container.
exec Execute a command in a container.
port-forward Forward one or more local ports to a pod.
proxy Run a proxy to the Kubernetes API server
run Run a particular image on the cluster.
stop Deprecated: Gracefully shut down a resource by name or filename.
expose Take a replication controller, service or pod and expose it as a new Kubernetes Service
autoscale Auto-scale a replication controller

Same Pod in Kubernetes
$ kubectl create -f mypod.yaml
pod "mypod" created
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: webserver
image: ubuntu
workingDir: /tmp/mypod
command: ["python3", "-m", "http.server", "80"]
volumeMounts:
- name: htdocs
mountPath: /tmp/mypod
- name: busybox
image: sttts/busybox-curl
tty: true
stdin: true
volumeMounts:
- name: htdocs
mountPath: /tmp/mypod
volumes:
- name: htdocs
hostPath:
path: /tmp/mypod

Documentation
Documentation landing page: http://kubernetes.io/v1.1

Documentation - Schema
JSON/YAML/REST schema: http://kubernetes.io/v1.1/docs/api-reference/v1/definitions.html

Examples – learn from them
Deployment examples: https://github.com/kubernetes/kubernetes/tree/master/examples

Browsing the API
Open https://192.168.65.65:6443/swagger-ui/

The source of truth: api/v1/types.go

On the wire (JSON only)
$ curl -k -u admin:admin
https://192.168.65.39:6443/api/v1/namespaces/default/pods/mypod
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "mypod",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/pods/mypod",
"uid": "37d657dc-b47a-11e5-938a-0242c0a84127",
"resourceVersion": "5792",
"creationTimestamp": "2016-01-06T13:34:32Z",
"annotations": {
"k8s.mesosphere.io/bindingHost": "192.168.65.44",
"k8s.mesosphere.io/executorId": "cf3cd4adb282b475_k8sm-executor",
"k8s.mesosphere.io/offerId": "20160106-081826-641837248-5050-1-O3680",
"k8s.mesosphere.io/slaveId": "20160106-081826-641837248-5050-1-S1",
"k8s.mesosphere.io/taskId": "pod.380df08e-b47a-11e5-8201-0242c0a8412b"
}
},
"spec": {
"volumes": [
{
"name": "htdocs",
"hostPath": {
"path": "/tmp/mypod"
}
},
{
"name": "default-token-b3a3b",
"secret": {

● in metadata block of every API resource
● labels can be used to filter objects serverside (annotations can’t)
$ kubectl get nodes -l gen=2011
Labels & Annotations
"annotations": {
"k8s.mesosphere.io/bindingHost": "192.168.65.44",
"k8s.mesosphere.io/executorId": "k8sm-executor",
...
}
for tooling
"labels": {
"git": "cb52d79578b1379e32b7e6a119ed16232ef1b13b",
"env": "prod",
"sla": "super-premium",
"app": "webshop",
"tier": "frontend"
}
for
user/admin

Getting the Pod IP
$ kubectl get pod mypod -o yaml
…
hostIP: 192.168.65.68
phase: Running
podIP: 192.168.66.11
startTime: 2016-01-07T08:33:55Z
$ kubectl get pod mypod -o template --template='{{.status.podIP}}'
192.168.66.12

From Pod to Pod
consumer
192.168.66.42
busybox
mypod with IP
192.168.66.6
webserver:80
busybox
pause
pause
192.168.66.6:80
mypod2 with IP
192.168.66.8
webserver:80
busybox
pause
192.168.66.8:80
● cluster routable IPs
● native ports without conflicts
● no “pod” service discovery available

A Service
consumer
192.168.66.42
busybox
mypod with IP
192.168.66.6
webserver:80
busybox
pause
pause
10.10.10.7:80
mypod2 with IP
192.168.66.8
webserver:80
busybox
pause
10.10.10.7:80
● a service as a static API object
$ kubectl create -f service.yaml
● virtual, but static IP (usually 10.x.y.z)
● no service discovery necessary
Service“mypods”withvirtualIP10.10.10.7
endpoint for
mypod
endpoint for
mypod2
192.168.66.8:80
192.168.66.6:80
brilliant

Defining a Service
$ cat mypods-service.yaml
apiVersion: v1
kind: Service
metadata:
name: mypods
spec:
ports:
- port: 80
selector:
name: mypod
$ kubectl create -f mypods-service.yaml
service "mypods" created
$ kubectl get services
NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE
k8sm-scheduler 10.10.10.34 <none> 10251/TCP <none> 6h
kubernetes 10.10.10.1 <none> 443/TCP <none> 6h
mypods 10.10.10.32 <none> 80/TCP name=mypods 5m
● match pods with these labels
● a service without any pod is completely valid
● services exist on their own

A Service, available on each node
consumer
192.168.66.42
busybox
mypod with IP
192.168.66.6
webserver:80
busybox
pause
pause
10.10.10.7:80
mypod2 with IP
192.168.66.8
webserver:80
busybox
pause
10.10.10.7:80
endpoint for
mypod
endpoint for
mypod2
192.168.66.8:80
192.168.66.6:80
● Implemented by kube-proxy
○ in user-mode (simple proxy)
○ or using iptables
● on every node!
traffic from a pod is
intercepted on the host of the
pod, not the destination of the
connection.
!
node A
Note: In addition kube-proxy makes
services available for other hosts, e.
g. for internet clients

mypod with IP
192.168.66.6
webserver:80
busybox
pause
mypod2 with IP
192.168.66.8
webserver:80
busybox
pause
endpoint for
mypod
endpoint for
mypod2
192.168.66.8:80
192.168.66.6:80
● 1 endpoint for each matching pod of a service
● endpoint = pod-IP:port, e.g. 192.168.66.6:80
$ kubectl get endpoints kube-ui
NAME ENDPOINTS AGE
mypods 192.168.65.68:8001 2s
Service Endpoints
Special in Kuberentes-Mesos because it
runs without overlay network here. Otherwise, :80
would be visible here.

Behind the scenes – Linux iptables
$ iptables -t nat -n -L
...
Chain KUBE-PORTALS-CONTAINER (1 references)
target prot opt source destination
REDIRECT tcp -- 0.0.0.0/0 10.10.10.3 /* kube-system/kube-ui: */ tcp dpt:80 redir ports 45123
REDIRECT tcp -- 0.0.0.0/0 10.10.10.181 /* default/k8sm-scheduler: */ tcp dpt:10251 redir ports 59722
REDIRECT tcp -- 0.0.0.0/0 10.10.10.1 /* default/kubernetes: */ tcp dpt:443 redir ports 55836
REDIRECT udp -- 0.0.0.0/0 10.10.10.10 /* kube-system/kube-dns:dns */ udp dpt:53 redir ports 57016
REDIRECT tcp -- 0.0.0.0/0 10.10.10.10 /* kube-system/kube-dns:dns-tcp */ tcp dpt:53 redir ports 42340

● state consists of all API objects, stored on the API-Server
● core Kubernetes architecture principle:
logic is in control loops (“controllers”) which
● are state-less (can recover from failure)
● and decoupled (communication via API-Server)
State plus Control
read state
process
state
write state

A Control Loop to run Pods
INPUT:
REPLICAS: number of mypods that should be running
ALGORITHM
1. running := number of running pods with label name=mypod
2. If running > REPLICAS => delete some mypods
3. If REPLICAS > running => launch some mypods
4. Goto 1

REPLICAS = int(sys.argv[1])
master = os.getenv("KUBERNETES_MASTER", "http://localhost:8080")
kubeclient = toolkit.KubeHTTPClient(api_server=master, debug=False)
while True:
# count mypods
response = kubeclient.execute_operation(method="GET",
ops_path="/api/v1/namespaces/default/pods?labelSelector=name%3Dmypod")
mypods = response.json()['items']
running_mypods = len(mypods)
print "{} running".format(running_mypods)
...
Getting # of running mypods

if running_mypods > REPLICAS:
to_delete = running_mypods - REPLICAS
print " Too many are running. Deleting {} pods:".format(to_delete)
for pod in mypods[:to_delete]:
print " Deleting pod {}".format(pod['metadata']['name'])
kubeclient.delete_resource(pod['metadata']['selfLink'])
If running > REPLICAS => delete some mypods

elif REPLICAS > running_mypods:
to_launch = REPLICAS - running_mypods
for n in range(0, to_launch):
mypod_spec, _ = util.load_yaml(filename="mypod.yaml")
mypod_spec["metadata"]["name"] += "-" + shortuuid.uuid()[:4].lower()
print "Launching pod {}".format(mypod_spec["metadata"]["name"])
response = kubeclient.execute_operation(method='POST',
ops_path = "/api/v1/namespaces/default/pods",
payload = util.serialize_tojson(mypod_spec))
If REPLICAS > running => launch some mypods

apiVersion: v1
kind: ReplicationController
metadata:
name: mypod
spec:
replicas: 10
selector:
name: mypod
template:
metadata:
labels:
name: mypod
spec:
containers:
- name: webserver
image: ubuntu
...
$ kubectl create -f mypod-rc.yaml
$ kubectl scale --replicas=5 rc/mypod
Kubernetes Replication Controllers – “RCs”

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: mypod
spec:
selector:
name: mypod
template:
metadata:
labels:
name: mypod
spec:
nodeSelector:
env=dmz
containers:
- name: webserver
image: ubuntu
...
● DaemonSets run pods once per node
● MyCassandraController might run Cassandra instances
● MyGaleraController might run MySQL Galera instances
● MyHaproxyUpdateController might write haproxy.conf
● MyNodeUpdateController might do rolling security
updates on nodes or of containers
● MyBitcoinScheduler might schedule bitcoin pods for
spare resources
Other Controllers

Controllers, controllers everywhere ...
etcd apiserver
controller-manager
scheduler
node controller
endpoint controller
kube-proxy
kube-proxy
kube-proxy
kubelet
kubelet
kubelet
replication controller
daemonset controller
service controller
token controller
resource quota
controller
persistent volume
controller
service account
controller
...

Of course: we are hiring in Hamburg, Berlin & San Francisco!

Backup

$ kubectl get pods -l "name=mypod" -w --v=9
=> GET https://192.168.65.39:6443/api/v1/watch/namespaces/default/pods?
labelSelector=name%3Dmypod
{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1", ...}}
{"type":"MODIFIED","object":{"kind":"Pod","apiVersion":"v1", …}}
{"type":"DELETED","object":{"kind":"Pod","apiVersion":"v1",...}}
Watches

Kubernetes Architecture and Introduction

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to Kubernetes Architecture and Introduction (20)

More from Stefan Schimanski (15)

Recently uploaded (20)

Kubernetes Architecture and Introduction