Linux Performance Profiling and Monitoring

Linux Performance Profiling and
Monitoring
Georg Schönberger

2
Thomas-Krenn.AG
_ A server manufacturer in
Bavaria
_ Well visited knowledge
base, Thomas-Krenn
Wiki

3
Agenda
_ Collect Statistics
_ Sysstat Package
_ dstat
_ nicstat
_ /proc → raw counters
_ sar and sadc
_ Watch online
_ top
_ htop
_ iotop
_ iftop
_ Tracing
_ perf_events
_ ftrace
_ perf-tools
_ Flame graphs
_ LXC
_ MySQL
_ Slow Query Log
_ innotop

# find / -type f -name statistics

From Brendan Gregg, http://www.brendangregg.com/linuxperf.html

8
mpstat
_ Without Interval/Count → since system startup
_ CPU usage per Core
_ Including Hyperthreading
_ Check how well usage is balanced
# mpstat -P ALL
Linux 3.13.0-48-generic (X220) 2015-04-14 _x86_64_ (4 CPU)
14:28:21 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
14:28:21 all 11,59 0,09 3,62 0,03 0,00 0,04 0,00 0,00 0,00 84,64
14:28:21 0 6,45 0,05 1,87 0,04 0,00 0,07 0,00 0,00 0,00 91,53
14:28:21 1 16,44 0,11 5,56 0,01 0,00 0,00 0,00 0,00 0,00 77,89
14:28:21 2 17,15 0,14 5,55 0,03 0,00 0,05 0,00 0,00 0,00 77,08
14:28:21 3 16,27 0,11 4,89 0,01 0,00 0,02 0,00 0,00 0,00 78,70
# lscpu | grep -E 'core|socket'
Thread(s) per core: 2
Core(s) per socket: 2

9
mpstat
# mpstat -P ALL 1 2
15:24:45 all 5,21 0,00 7,12 17,81 0,00 0,27 0,00 0,00 0,00 69,59
15:24:45 0 1,43 0,00 1,43 0,00 0,00 2,86 0,00 0,00 0,00 94,29
15:24:45 1 11,88 0,00 23,76 64,36 0,00 0,00 0,00 0,00 0,00 0,00
15:24:45 2 4,12 0,00 1,03 0,00 0,00 0,00 0,00 0,00 0,00 94,85
15:24:45 3 3,03 0,00 1,01 0,00 0,00 0,00 0,00 0,00 0,00 95,96
15:24:46 all 5,74 0,00 7,10 17,76 0,00 0,55 0,00 0,00 0,00 68,85
15:24:46 0 2,99 0,00 1,49 0,00 0,00 2,99 0,00 0,00 0,00 92,54
15:24:46 1 11,88 0,00 23,76 64,36 0,00 0,00 0,00 0,00 0,00 0,00
15:24:46 2 6,00 0,00 1,00 0,00 0,00 0,00 0,00 0,00 0,00 93,00
15:24:46 3 1,01 0,00 1,01 0,00 0,00 0,00 0,00 0,00 0,00 97,98
Core 1 is not idle
and also deals
with %iowait

10
vmstat
_ High Level Statistics about
_ Virtual memory
_ Swap/Paging
_ I/O statistics
_ System interrupts and context switches
_ CPU statistics
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 0 172 371856 137088 3125664 0 0 0 153060 7618 7059 17 9 56 17 0
3 0 172 416596 137096 3125704 0 0 0 163420 8689 7419 11 10 61 17 0
0 0 172 451716 137096 3089916 0 0 0 0 396 1848 3 1 96 0 0
0 0 172 413916 137108 3118796 0 0 0 52 502 2218 9 2 90 0 0
2 0 172 399756 137108 3118860 0 0 284884 0 14830 10941 10 13 66 12 0
1 1 172 364948 137108 3118988 0 0 310792 0 16204 12738 20 13 53 14 0

11
vmstat
_ Memory statistics
_ buff Raw disk blocks like filesystem metadata
_ cache Memory used for data information, pages with actual
contents
$ vmstat 1
0 0 172 607760 182172 3313684 0 0 159 496 154 222 18 6 76 0 0
0 0 172 607628 182172 3313684 0 0 0 52 387 2008 4 2 95 0 0
0 0 172 607348 182172 3313684 0 0 0 0 397 2034 4 1 95 0 0
0 0 172 606448 182172 3313684 0 0 0 0 378 1896 4 2 94 0 0
$ free
total used free shared buffers cached
Mem: 8056664 7450316 606348 491820 182172 3313684
-/+ buffers/cache: 3954460 4102204
Swap: 1048572 172 1048400

12
vmstat
_ Process related fields
_ r The number of runnable processes (running or waiting for
run time)
_ If high → indicator for saturation
_ b The number of processes in uninterruptible sleep
_ Mostly waiting for I/O
# vmstat 1
[...]
0 1 172 404768 137088 3125664 0 0 0 167524 9029 6955 6 6 70 18 0
0 1 172 399956 137088 3125664 0 0 0 138340 8133 6165 7 7 68 19 0
$ ps -eo ppid,pid,user,stat,pcpu,comm,wchan:32 | grep ext4
[...]
7159 7161 root Ds 3.2 fio ext4_file_write
Kernel function process
is sleeping on
Processes doing I/O
can bei in waiting state

13
vmstat plots
https://clusterbuffer.wordpress.com/admin-tools/vmstat-plotter/
Drawing interrupts
and context switches

But we are not satisfied
with summaries and overviews...

16
pidstat
_ Report statistics for tasks being managed by
kernel
_ CPU bound → identify peak activity
$ top -b -n 1 -d 2 -o %CPU | head
[...]
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9059 gschoenb 20 0 47532 21132 2444 R 96,9 0,3 0:02.14 python
1 root 20 0 33880 3256 1500 S 0,0 0,0 0:02.35 init
$ pidstat -p 9059 -u 1 -l
10:11:04 UID PID %usr %system %guest %CPU CPU Command
10:11:05 1000 9059 100,00 0,00 0,00 100,00 0 python ijk-matrix.py
-i matrix.in
-i matrix.in
-i matrix.in
Even check command
line arguments!

17
pidstat
_ I/O bound → device report
# mpstat -P ALL 1
10:25:32 all 14,88 0,00 9,40 13,84 0,00 1,04 0,00 0,00 0,00 60,84
10:25:32 0 22,45 0,00 1,02 0,00 0,00 0,00 0,00 0,00 0,00 76,53
10:25:32 1 13,73 0,00 34,31 51,96 0,00 0,00 0,00 0,00 0,00 0,00
10:25:32 2 17,86 0,00 0,00 0,00 0,00 3,57 0,00 0,00 0,00 78,57
10:25:32 3 6,12 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 93,88
# pidstat -d 1
10:26:35 UID PID kB_rd/s kB_wr/s kB_ccwr/s Command
10:26:36 0 9208 0,00 2303,85 0,00 fio
10:26:36 0 9209 0,00 2996,15 0,00 fio
10:26:36 0 9210 0,00 2023,08 0,00 fio
10:26:36 0 9211 0,00 1284,62 0,00 fio
Which process
is causing %iowait?
Device report reveals
command and I/O

18
pidstat
_ How much memory is PID 8461 using?
_ Major faults require I/O operations, good indicator you need
more RAM!
_# pidstat -r -p 8461 1 3
10:09:06 UID PID minflt/s majflt/s VSZ RSS %MEM Command
10:09:07 1000 8461 8,00 0,00 2018384 786688 9,76 firefox
10:09:08 1000 8461 11,00 0,00 2018384 786688 9,76 firefox
10:09:09 1000 8461 23,00 0,00 2018448 786892 9,77 firefox
Average: 1000 8461 14,00 0,00 2018405 786756 9,77 firefox
Current used share
of physical memory
Minor and major
page faults

19
iostat
_ I/O subsystem statistics
_ CPU or device utilization report
_ Without argument → summary since boot
_ Skip that with -y option
# iostat
avg-cpu: %user %nice %system %iowait %steal %idle
16,16 0,09 4,79 0,46 0,00 78,50
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 83,80 41,64 531,43 22375057 285581196

20
iostat
_ CPU util report → %iowait
_ Not really reliable → %iowait is some kind of
%idle time
# taskset 1 fio –rw=randwrite [...] &
# iostat -y -c 1 3
[…]
17,32 0,00 6,56 13,65 0,00 62,47
# taskset 1 sh -c "while true; do true; done" &
# iostat -y -c 1 3
35,59 0,00 7,02 0,00 0,00 57,39
http://www.percona.com/blog/2014/06/03/trust-vmstat-iowait-numbers/

21
iostat
_ Extended device util report → %util
_ man iostat → … for devices serving requests in parallel, such as
RAID arrays and modern SSDs, this number does not reflect
their performance limits.
_ In theory
_ 94,4% util 23032 IOPS
_ 99,6% util 24300 IOPS

22
iostat
# iostat -y -d -x 1 3
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await
r_await w_await svctm %util
sda 0,00 2,00 0,00 23032,00 0,00 92136,00 8,00 2,90 0,13
0,00 0,13 0,04 94,40
# iostat -y -d -x 1 3
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await
r_await w_await svctm %util
sda 0,00 2917,00 0,00 43175,00 0,00 184500,00 8,55 135,75 3,15
0,00 3,15 0,02 99,60
Only 5% util increase,
but IOPS nearly doubled!

23
https://www.thomas-krenn.com/de/wiki/Linux_Storage_Stack_Diagramm

24
iostat
_ avgqu-sz Avg. queue length of requests issued
_ (delta[time_in_queue] / interval) / 1000.0
_ time_in_queue Requets waiting for device, effected by in_flight
_ await Avg. time requests being served
_ delta[read_ticks + write_ticks] / delta[read_IOs +
write_Ios]
_ ticks also effected by in_flight
_ Therefore serving more requests while await is
not increasing, is a good performance indicator
- Monitoring IO Performance using iostat and pt-diskstats
- Block layer statistics

26
dstat
_ Combines several classic tools
_ Prints metrics and uses colors
_ Has a plugin concept

27
nicstat
_ Print network device statistics
_ %Util depends on speed and duplex mode
_ Sat also takes errors into account
# nicstat -l
Int Loopback Mbit/s Duplex State
vboxnet0 No 0 unkn up
eth0 No 1000 full up
lo Yes - unkn up
wlan0 No 0 unkn up
# nicstat -i eth0 1 5
Time Int rKB/s wKB/s rPk/s wPk/s rAvs wAvs %Util Sat
14:52:21 eth0 3.08 0.36 3.13 2.48 1007.6 149.4 0.00 0.05
14:52:22 eth0 19.89 1.23 16.98 17.97 1199.6 70.00 0.02 0.00
14:52:23 eth0 21.42 1.09 21.99 16.00 997.1 70.00 0.02 0.00
# nicstat -i eth0 -t 1 2
14:57:36 InKB OutKB InSeg OutSeg Reset AttF %ReTX InConn OutCon Drops
TCP 0.00 0.00 2.88 2.51 0.02 0.00 0.000 0.00 0.04 0.00
14:57:37 InKB OutKB InSeg OutSeg Reset AttF %ReTX InConn OutCon Drops
TCP 0.00 0.00 0.00 0.00 0.00 0.00 0.000 0.00 0.00 0.00
Check if your network is
saturated, Drops can be
an indicator!

Do you have a basic chronicle
of your system's data?
Yes No

29
sar
_ It's easy with system activity reporter
_ sar, sadc, sa1 and sa2, sadf
http://www.brendangregg.com/Perf/linux_observability_sar.png

30
ksar
_ LC_ALL=POSIX sar -A -f sa10 > ksar.out.txt
Mitigates character encoding
and number format problems

31
The /proc filesystem
_ /proc knows everything
$ sudo procinfo -n 2
Memory: Total Used Free Buffers
RAM: 8056664 7769252 287412 161476
Swap: 1048572 172 1048400
Bootup: Thu Apr 9 07:16:14 2015 Load average: 0.69 0.47 0.47 1/600 1711
user : 05:45:48.75 11.7% page in : 4844721
nice : 00:02:35.09 0.1% page out: 22309056
system: 01:48:13.52 3.6% page act: 3638373
IOwait: 00:00:48.03 0.0% page dea: 799382
hw irq: 00:00:00.19 0.0% page flt: 112500382
sw irq: 00:01:18.48 0.0% swap in : 0
idle : 1d 17:48:14.29 84.5% swap out: 44
uptime: 5d 06:59:42.28 context : 247128528
[...]

32
Overview
_ Sysstat Package
_ dstat
_ nicstat
_ sar and sadc
_ Watch online
_ top
_ htop
_ iotop
_ iftop
_ Tracing
_ perf_events
_ ftrace
_ perf-tools
_ Flame graphs
_ strace
_ LXC
_ MySQL
_ Slow Query Log
_ innotop

33
top
_ System summary at beginning
_ Per process metrics afterwards
_ Default sorted by CPU usage
$ top -b -n 1| head -15
top - 15:33:50 up 3 days, 19:02, 3 users, load average: 0.13, 0.51, 0.59
Tasks: 668 total, 1 running, 667 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.5%us, 0.3%sy, 0.1%ni, 98.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 132009356k total, 23457172k used, 108552184k free, 1600120k buffers
Swap: 3904444k total, 0k used, 3904444k free, 12682188k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29276 root 20 0 6928 3488 668 S 19 0.0 22:55.72 ossec-syscheckd
1193 gschoenb 20 0 17728 1740 936 R 4 0.0 0:00.02 top
11257 root 20 0 22640 2636 1840 S 4 0.0 70:38.88 openvpn
19907 www-data 20 0 197m 61m 52m S 4 0.0 0:06.18 apache2
775 root 20 0 0 0 0 S 2 0.0 8:03.13 md3_raid10
3712 root 39 19 0 0 0 S 2 0.0 22:45.85 kipmi0
12807 root -3 0 0 0 0 S 2 0.0 6:20.30 drbd2_asender
18653 root 20 0 0 0 0 S 2 0.0 12:40.19 drbd1_receiver
1, 5 and 15 min
load average

34
top
_ Memory usage
_ VIRT The total size of virtual memory for the process
_ Also including e.g. not already mapped heap or swap
_ RES How many blocks are really allocated and mapped to
address space → resident
_ Also includes file-backed memory (like shared libraries, mmap)
_ Can be used concurrently by processes
_ SHR is shared or file-backed memory
_ RES – SHR = anon mem (malloc)
- https://www.linux.com/learn/tutorials/42048-uncover-the-meaning-of-tops-statistics
- http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html
$ cat /proc/17692/statm
1115764 611908 16932 26 0 848936 0

35
top
_ Can consume resources on it's own
_ Toggle f and select fields, e.g. SWAP
_ -u let's you see processes from a user
_ Toggle k to kill a PID
_ Toggle r to renice a PID
_ But
_ top can miss short living processes
_ high %CPU → so what?
_ Keep an eye on the tracing part

36
htop
_ „Super advanced“ top
_ Uses colors, views can be customized

37
iotop
_ Simple top like I/O monitor
_ Which process is causing I/O
_ Filtering specific PID is possible
# iotop -o -b
Total DISK READ : 0.00 B/s | Total DISK WRITE : 63.94 M/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 63.90 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
19153 be/4 root 0.00 B/s 63.89 M/s 0.00 % 75.44 % fio --rw=randwrite --name=test
--filename=test.fio --size=300M --direct=1 --bs=4k
17715 be/4 gschoenb 0.00 B/s 46.18 K/s 0.00 % 0.00 % firefox [mozStorage #1]
# iotop -o -b
Total DISK READ : 69.02 M/s | Total DISK WRITE : 65.92 K/s
Actual DISK READ: 69.02 M/s | Actual DISK WRITE: 345.12 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND
19176 be/4 root 69.02 M/s 0.00 B/s 0.00 % 88.28 % fio --rw=read --name=test
--filename=test.fio --size=300M --direct=1 --bs=8k
Show writes, reads
and command in
realtime

38
Bandwidth live usage
_ iftop
_ Per interface usage
_ nethogs
_ Per proces
NetHogs version 0.8.0
PID USER PROGRAM DEV SENT RECEIVED
17692 gschoenb /usr/lib/firefox/firefox eth0 0.162 0.194 KB/sec
16585 root /usr/bin/ssh eth0 0.000 0.000 KB/sec
16611 gschoenb evolution eth0 0.000 0.000 KB/sec
? root unknown TCP 0.000 0.000 KB/sec
TOTAL 0.162 0.194 KB/sec

40
Overview
_ Sysstat Package
_ dstat
_ nicstat
_ sar and sadc
_ Watch online
_ top
_ htop
_ iotop
_ iftop
_ Tracing
_ perf_events
_ ftrace
_ perf-tools
_ Flame graphs
_ LXC
_ MySQL
_ Slow Query Log
_ innotop

42
Profiling
_ Create profile about usage characteristics
_ Count specific samples/events
_ Count objects
_ Next slides focus on system profiling
_ ftrace
_ perf_events and perf
_ Collecting statistics about tracepoints
_ Lines of kernel code with defined event

43
ftrace
_ Part of the Linux kernel since 2.6.27 (2008)
_ What is going on inside the kernel
_ Common task is to trace events
_ With ftrace configured, only debugfs is
required
# cat /proc/sys/kernel/ftrace_enabled
1
# mount | grep debug
none on /sys/kernel/debug type debugfs (rw)
/sys/kernel/debug/tracing# cat available_tracers
blk mmiotrace function_graph wakeup_rt wakeup function nop

44
ftrace
_ Interact with files in sys
_ Easier with trace-cmd
#!/bin/bash
DEBUGFS=`grep debugfs /proc/mounts | awk '{ print $2; }'`
echo $$ > $DEBUGFS/tracing/set_ftrace_pid
echo function > $DEBUGFS/tracing/current_tracer
echo 1 > $DEBUGFS/tracing/tracing_on
exec $*
echo 0 > $DEBUGFS/tracing/tracing_on

45
perf_events and perf
_ Used to be called performance counters for
Linux
_ A lot of updates for kernel 4.1
_ https://lkml.org/lkml/2015/4/14/264
_ CPU performance counters, tracepoints,
kprobes and uprobes
_ Per package with linux-tools-common
# which perf
/usr/bin/perf
# dpkg -S /usr/bin/perf
linux-tools-common: /usr/bin/perf

46
perf list
_ perf list
_ Shows supported events
# perf list | wc -l
1779
# perf list | grep Hardware
cpu-cycles OR cycles [Hardware event]
instructions [Hardware event]
cache-references [Hardware event]
cache-misses [Hardware event]
branch-instructions OR branches [Hardware event]
branch-misses [Hardware event]
bus-cycles [Hardware event]
stalled-cycles-frontend OR idle-cycles-frontend [Hardware event]
stalled-cycles-backend OR idle-cycles-backend [Hardware event]
ref-cycles [Hardware event]
L1-dcache-loads [Hardware cache event]
L1-dcache-load-misses [Hardware cache event]
L1-dcache-stores [Hardware cache event]
L1-dcache-store-misses [Hardware cache event]
This also includes
static tracepoints

47
Raw CPU counters
_ Each CPU has it's own raw counters
_ They should be documented by the hardware manufacturer
_ https://download.01.org/perfmon/
_ libpfm4 is a nice way to find raw masks
# perf list | grep rNNN
rNNN [Raw hardware event descriptor]
# git clone git://perfmon2.git.sourceforge.net/gitroot/perfmon2/libpfm4
# cd libpfm4
# make
# cd examples/
# ./showevtinfo | grep LLC | grep MISSES
Name : LLC_MISSES
[...]
# ./check_events LLC_MISSES | grep Codes
Codes : 0x53412e
# perf stat -e r53412e sleep 5
Now we collect last
level cache misses
with the raw mask

48
Tracepoints
_ perf also has trace functionalities
_ Filesystem
_ Block layer
_ Syscalls
# perf list | grep -i trace | wc -l
1716
# perf stat -e 'syscalls:sys_enter_mmap' ./helloWorld.out
Hello world!
Performance counter stats for './helloWorld.out':
8 syscalls:sys_enter_mmap
0,000556961 seconds time elapsed

49
perf stat
_ Get a counter summary
# perf stat python numpy-matrix.py -i matrix.in
Performance counter stats for 'python numpy-matrix.py -i matrix.in':
576,104221 task-clock (msec) # 0,930 CPUs utilized
319 context-switches # 0,554 K/sec
4 cpu-migrations # 0,007 K/sec
9.738 page-faults # 0,017 M/sec
1.743.664.199 cycles # 3,027 GHz [82,63%]
831.364.029 stalled-cycles-frontend # 47,68% frontend cycles idle [83,75%]
458.760.523 stalled-cycles-backend # 26,31% backend cycles idle [67,26%]
2.793.953.303 instructions # 1,60 insns per cycle
# 0,30 stalled cycles per insn [84,28%]
573.342.473 branches # 995,206 M/sec [83,78%]
3.586.249 branch-misses # 0,63% of all branches [82,70%]
0,619482128 seconds time elapsed
Easy to compare
performance of
different algorithms

50
perf record
_ Record samples to a file
_ Can be post-processed with perf report
_ -a records on all CPUs
_ -g records call graphs
_ Install debug symbols
# perf record -a -g sleep 5
[ perf record: Woken up 4 times to write data ]
[ perf record: Captured and wrote 2.157 MB perf.data (~94254 samples) ]
Nice way to record
what's currently
running on all CPUs

51
perf report
_ Displays profile of a record
_ Can be sorted and or filtered
_ Shows all samples

53
perf-tools
_ By Brendan Gregg
_ https://github.com/brendangregg/perf-tools
_ Mostly quick hacks, read Warnings!
_ Using perf_events and ftrace
_ Good examples what can be done with perf and
ftrace
_ iosnoop Shows I/O access for commands, including latency
_ cachestat Linux page cache hit/miss statistics
_ functrace Count kernel functions matching wildcards
Nice, this are simple
bash scripts!

56
Flamegraph
_ Visualization how resources are distributed
among code
Powered by @agentzh, http://agentzh.org/misc/slides/yapc-na-2013-flame-graphs.pdf

57
Flamegraph
# perf record -g dd if=/dev/zero of=test.data count=1 bs=1M
# mv perf.data perf.data.dd
# perf script -i perf.data.dd | ./FlameGraph/stackcollapse-perf.pl > out.dd.folded
# ./FlameGraph/flamegraph.pl out.dd.folded > out.perf.dd.svg
Flame Graph
__do_page..
ext..
_aesni_enc1
ecryptfs_encrypt_page
do_sync_write
do_page_f..
sys_execve
__a..
do_..
dd
unm..
ext..
__alloc_p..
load_elf_binary
generic_file_aio_write
__b..
ecr..
generic_file_buffered_write
read_zero..
__mark_in..
do_..
cle..
generic_file_buffered_write
rw_..
__r..
vfs_write
intel_pmu_enable_all
khu..
sys_write
ecryptfs_..
__GI___libc_write
handle_mm..
mem..
ablk_encrypt
ext..
__e..
alloc_pag..
__ablk_encrypt
__generic_file_aio_write
pag..
sys..
page_fault
x86_pmu_enable
mut..
ecryptfs_write_lower
get..
ext..
ext..
blo..
_co..
han..
gra..
system_ca..
kernel_write
ecryptfs_..
kernel_wr..
generic_write_..
sec..
sca..
native_write_msr_safe
perf_event_comm
sea..
do_sync_write
do_..
mmput
ext4_file_write
[unknown]
jbd..
_nl..
sys..
loa..
sys..
cop..
stu..
perf_pmu_enable
generic_file_aio_write
perf_event_context_sched_in
ext..
__GI___li..
per..
ext4_da_w..
set_task_comm
__d..
vfs_write
crypt_extent
unm..
[unknown]
setup_new_exec
all..
pag..
set..
sys_read
app..
fin..
:29802
do_..do_..
vfs_write
ext4_dirt..
__e..
_co..
crypt_scatterlist
vfs_read
ext4_da_write_..
__b..
cbc..
fil..
do_execve_common.isra.22
unm..
read_zero
stub_execve
__d..
_dl..
exi..
exp..
com..
__p..
search_binary_handler
ecryptfs_write_end
system_call_fastpath
__generic_file_aio_write
blk..

59
Linux Container
_ Lightweight „virtual machines“ using features
provided by a modern Linux kernel
_ cgroups Aggregate or partition tasks and their children to
hierarchical groups to isolate resources
_ namespaces Wrap a resource in an abstraction so that it
appears to processes they have their own isolated resource
_ Each container shares the kernel running on
the host
_ Some may refer to it as „native performance“

60
Linux Container
_ cgroups are divided into subsystems, e.g.
_ cpusets
_ blkio
_ memory
Image from Boden Russel, http://de.slideshare.net/BodenRussell/realizing-linux-containerslxc

61
Linux Container
_ cgroup created per container per subsystem
# lxc-ls --fancy
NAME STATE IPV4 IPV6 GROUPS AUTOSTART
-----------------------------------------------------
ubuntu1 RUNNING 10.0.3.119 - - NO
# lxc-info -n ubuntu1
Name: ubuntu1
State: RUNNING
PID: 7548
IP: 10.0.3.119
CPU use: 1.80 seconds
BlkIO use: 22.68 MiB
Memory use: 30.85 MiB
KMem use: 0 bytes
Link: vethC8TJUT
TX bytes: 3.33 KiB
RX bytes: 3.49 KiB
Total bytes: 6.82 KiB

62
Linux Container
_ lxc-info takes cgroups into account
_ cgroups providing further info
_ memory.stat
_ memory.failcnt
_ cpuset.cpus
Value Origin
CPU use lxc-cgroup -n ubuntu1 cpuacct.usage
BlkIO use lxc-cgroup -n ubuntu1 blkio.throttle.io_service_bytes
Memory use lxc-cgroup -n ubuntu1 memory.usage_in_bytes
KMem use lxc-cgroup -n ubuntu1 memory.kmem.usage_in_bytes
Link cat /sys/class/net/veth0EP3QM/statistics/*_bytes

63
Linux Container
_ lxc-top monitors container overall usage
_ Traditional tools without lxcfs do not work!
# lxc-top
Container CPU CPU CPU BlkIO Mem
Name Used Sys User Total Used
ubuntu1 1.94 1.11 0.84 32.22 MB 14.71 MB
ubuntu2 1.43 0.88 0.79 10.61 MB 17.88 MB
TOTAL 2 of 2 3.36 1.99 1.63 42.83 MB 32.59 MB
root@host # lxc-cgroup -n ubuntu1 memory.limit_in_bytes
33554432
root@container # free -h
total used free shared buffers cached
Mem: 489M 202M 287M 488K 26M 115M

# sh -c "while true; do mysql; done"

65
MySQL
_ Percona provides a lot of good tools
_ First step, generate a summery
# pt-mysql-summary
# Percona Toolkit MySQL Summary Report #######################
# Instances ##################################################
Port Data Directory Nice OOM Socket
===== ========================== ==== === ======
/var/lib/mysql 0 0 /var/lib/mysql/mysql.sock
# MySQL Executable ###########################################
Path to executable | /usr/sbin/mysqld
Has symbols | Yes
# Report On Port 3306 ########################################
User | root@localhost
Time | 2015-04-14 07:49:09 (CEST)
Hostname | mysql1
Databases | 15
Datadir | /var/lib/mysql/
[...]

66
MySQL
_ Extended status prints also counters
_ Can be monitored with pmp-check-mysql-status Plugin
_ Slow Query log
_ Queries exceeding a specific runtime
_ OFF by default, runtime and log file must be defined
_ Query Cache is ignored
# mysqladmin ext | wc -l
345
# mysqladmin ext | grep Threads_running
| Threads_running | 3
# mysqladmin ext | grep Innodb_buffer_pool_pages_free
| Innodb_buffer_pool_pages_free | 12298

67
MySQL
_ Easy way to log all queries → long_query_time 0
_ pt-query-digest
_ Process slow log and generate report
# pt-query-digest mysql-slow.log
[...]
# Attribute total min max avg 95% stddev median
# ============ ======= ======= ======= ======= ======= ======= =======
# Exec time 184984s 9s 419s 51s 151s 45s 42s
# Lock time 15s 0 3s 4ms 0 71ms 0
# Rows sent 500.05M 0 2.65M 139.79k 1.69M 491.22k 3.89
# Rows examine 3.23G 0 234.22M 923.81k 2.49M 5.53M 440.37k
# Query size 128.45M 6 2.75M 35.91k 68.96k 136.30k 10.29k
# Profile
# Rank Query ID Response time Calls R/Call V/M Item
# ==== ================== ================ ===== ======== ===== ==========
# 1 0x7A8EB8C13A4A8435 29885.0000 16.2% 305 97.9836 22.08 SELECT
# 2 0xA45C5FB6D066119B 26077.0000 14.1% 369 70.6694 22.49 SELECT
# 3 0x67A347A2812914DF 13737.0000 7.4% 397 34.6020 14.53 SELECT
# 4 0xD7A9797E81785092 11855.0000 6.4% 121 97.9752 22.05 SELECT

68
MySQL – innotop
_ Live analysis of SQL queries
_ Sort by execution time
_ Not only for innodb

Thanks for your attention!
_ gschoenberger@thomas-krenn.com
_ @devtux_at

71
iostat
_ CPU util report → %iowait
_ With iostat, avg. over CPUs
_ Take mpstat into account
# iostat -y -c 1 3
11,41 0,00 9,28 12,20 0,00 67,11
# mpstat -P ALL 1
12:40:48 all 10,08 0,00 7,36 16,08 0,00 0,82 0,00 0,00 0,00 65,67
12:40:48 0 13,68 0,00 25,26 61,05 0,00 0,00 0,00 0,00 0,00 0,00
12:40:48 1 18,81 0,00 0,99 0,00 0,00 0,00 0,00 0,00 0,00 80,20
12:40:48 2 1,43 0,00 1,43 0,00 0,00 4,29 0,00 0,00 0,00 92,86
12:40:48 3 4,04 0,00 1,01 0,00 0,00 0,00 0,00 0,00 0,00 94,95

72
The /proc filesystem
_ /proc knows everything
_ Obviously ps uses /proc
_ Or values from free
$ cat /proc/meminfo | grep Swap
SwapCached: 0 kB
SwapTotal: 1048572 kB
SwapFree: 1048400 kB
$ cat /proc/meminfo | grep -E 'Buffer|Cache'
Buffers: 183988 kB
Cached: 3365960 kB
SwapCached: 0 kB
$ cat /proc/self/status | head
Name:cat
State: R (running)
Pid: 13871
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000

73
varnishstat and top
_ Good example for application based statistics
# varnishstat -1 | wc -l
311
# # varnishstat -1 | grep -E 'cache|nuked'
MAIN.cache_hit 64616 1.58 Cache hits
MAIN.cache_hitpass 5775 0.14 Cache hits for pass
MAIN.cache_miss 38611 0.95 Cache misses
MAIN.n_lru_nuked 0 . Number of LRU nuked objects
# varnishtop -i ReqUR -1
558.00 ReqURL /de/wikiDE/api.php
384.00 ReqURL /favicon.ico
321.00 ReqURL /en/wikiEN/api.php
282.00 ReqURL /res/font/FSMeWeb/bold/fs_me_web-bold.woff
282.00 ReqURL /res/font/FSMeWeb/regular/fs_me_web-regular.woff
280.00 ReqURL /de/wikiDE/skins/tkskin/images/thomas-krenn-logo-grey.png
278.00 ReqURL /de/wikiDE/skins/tkskin/images/TK_Logo_200x90.png

74
blktrace
_ blktrace
_ Captures I/O traces
_ Trace is stored in a binary format
_ blkparse
_ Reads traces recorded by blktrace
_ btt
_ blktrace timeline
_ Post-processing tool

/var/log# head -n 1 syslog.1
# blktrace -d /dev/sda (run parallel)
# blkparse -i sda.blktrace.0[...]
8,0 0 1 0.000000000 610 A WS 228404656 + 8 <- (8,3) 226854320
8,0 0 2 0.000001180 610 Q WS 228404656 + 8 [kworker/0:3]
8,0 0 3 0.000006522 610 G WS 228404656 + 8 [kworker/0:3]
8,0 0 4 0.000009315 610 I WS 228404656 + 8 [kworker/0:3]
8,0 0 5 0.000012967 610 D WS 228404656 + 8 [kworker/0:3]
8,0 0 6 0.000043329 610 A WS 228404664 + 8 <- (8,3) 226854328
8,0 0 7 0.000043714 610 Q WS 228404664 + 8 [kworker/0:3]
8,0 0 8 0.000044562 610 G WS 228404664 + 8 [kworker/0:3]
8,0 0 9 0.000045143 610 I WS 228404664 + 8 [kworker/0:3]
8,0 0 10 0.000045701 610 D WS 228404664 + 8 [kworker/0:3]
[...]
8,0 3 1 0.431446519 5789 A R 425590688 + 32 <- (8,3) 424040352
8,0 3 2 0.431447779 5789 Q R 425590688 + 32 [head]
8,0 3 3 0.431452713 5789 G R 425590688 + 32 [head]
8,0 3 4 0.431454407 5789 P N [head]
8,0 3 5 0.431456930 5789 I R 425590688 + 32 [head]
8,0 3 6 0.431458376 5789 U N [head] 1
8,0 3 7 0.431461366 5789 D R 425590688 + 32 [head]
Device
CPU
Sequence
#
Tim
e
stam
p
PID
Action
Startblock
+
#
ofblocks
Process
Actions
A IO was remapped to a different device
Q IO handled by request queue code
G Get request
P Plug request
I IO inserted onto request queue
U Unplug request
D IO issued to driver RW
BS
RWBS
R read
W write
D block discard
B barrier operation
S synchronous operation

CPU0 (sda):
Reads Queued: 0, 0KiB Writes Queued: 7, 28KiB
Read Dispatches: 0, 0KiB Write Dispatches: 7, 28KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 0, 0KiB Writes Completed: 5, 20KiB
Read Merges: 0, 0KiB Write Merges: 0, 0KiB
Read depth: 1 Write depth: 7
IO unplugs: 0 Timer unplugs: 0
CPU1 (sda):
CPU3 (sda):
Total (sda):
Throughput (R/W): 37KiB/s / 74KiB/s
Events (sda): 62 entries
Skips: 0 forward (0 - 0.0%)
per CPU
details writes submitted
on this CPU
writes completed
on this CPU
avg. throughput
per device
details

# btt -i sda.blktrace.0
==================== All Devices ====================
ALL MIN AVG MAX N
--------------- ------------- ------------- ------------- -----------
Q2Q 0.000016944 0.000022114 0.000042534 6
Q2G 0.000000694 0.000001430 0.000005342 7
G2I 0.000000314 0.000000725 0.000002793 7
I2D 0.000000375 0.000000906 0.000003652 7
D2C 0.000992471 0.001018423 0.001048992 5
Q2C 0.000993887 0.001022085 0.001060779 5
[...]
D2C Driver and device time – the average time from when the actual
IO was issued to the driver until is completed (completion trace) back
to the block IO layer.
Q2C Measures the times for the complete life cycle of IOs during
the run.

78
MySQL pt-query-digest
# Query 1: 0.00 QPS, 0.01x concurrency, ID 0x67A347A2812914DF at byte 61989638
# This item is included in the report because it matches --limit.
# Scores: V/M = 172.18
# Time range: 2012-05-23 00:00:26 to 2015-04-17 00:10:33
# Attribute pct total min max avg 95% stddev median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count 12 11267
# Exec time 29 462888s 3s 2629s 41s 130s 84s 17s
# Lock time 0 531ms 0 599us 47us 93us 28us 38us
# Rows sent 99 82.85G 306 63.71M 7.53M 46.53M 13.84M 915.49k
# Rows examine 38 82.85G 306 63.71M 7.53M 46.53M 13.84M 915.49k
# Query size 0 683.69k 47 79 62.14 72.65 7.79 56.92
# String:
# Databases XXXXX (10159/90%)... 1 more
# Hosts localhost (9524/84%), XXXXX (1743/15%)
# Users XXXXX (9738/86%), XXXXX (1529/13%)
# Query_time distribution
# 100us
# 1ms
# 10ms
# 100ms
# 1s #################
# 10s+ ################################################################

79
MySQL
_ Performance Schema
_ Per default on in 5.6, older profiling commands are deprecated
with 5.6.7
_ A structured way in SQL to get timing information
_ Runtime and query execution statistics
_ Sys scchema (ps_helper) provides a more
comfortable way
> select * from schema_table_statistics
where table_schema='sbtest' limit 1 G
*************************** 1. row ***************************
table_schema: sbtest
table_name: sbtest
rows_fetched: 158764154
[...]

Linux Performance Profiling and Monitoring

More Related Content

What's hot (20)

Similar to Linux Performance Profiling and Monitoring (20)

Recently uploaded (20)

Linux Performance Profiling and Monitoring