Linux monitoring and Troubleshooting for DBA's

Linux Monitoring and Troubleshooting
By
MySQL Consulting
Team

About MyDBOPS
• MySQL Consulting
• MySQL Support
• Remote DBA support.
• Expert MySQL solutions
• MariaDB, Percona, Galera, TokuDB are
supported too.

Agenda
• Introduction
• Linux Architecture
• Devices we monitor
• Important tools
• Examples

INTRODUCTION
 Unix-like computer operating system.
 The monitoring and maintenance of the system is the most important task.
 There are many commands line utilities created for this purpose.
CPU l
Network Bandwidth
Memory usage
Disk usage
Syslogs & A
Running processes
etc.

Important tools
netstat
free
Df
Du
ps
history
lsof
top
systat
Sar
iostat
vmstat
Pidstat
mpstat
ping
telnet
ifconfig
w & uptime
who
whoami

Top
 Small tool, pre-installed in many unix systems.
 Display all running and active real time process
in ordered list & updates it regularly.
 CPU usage, Memory usage, Swap Memory, Cache Size,
Buffer Size, Process PID, User, Commands and much
more
 shows high memory and cpu utilization of a running
processes.

sysstat
 powerful logging and monitoring tool for Linux/Unix systems.
 contains utilities to monitor system performance and usage activity.
 used to monitor system performance and troubleshoot problems.
 sysstat is a go-to for power users and can log and track pretty much everything going on
within
your Linux box.
To install :
• redhat based - yum install sysstat
• Debian based - apt-get install sysstat

sar
 Sar(System Activity Report) is part of the sysstat package.
 collects and displays ALL system activities statistics.
 can monitor performance of various Linux subsystems (CPU, Memory, I/O..) in real time.
 also collect all performance data on an on-going basis, store them, and do historical analysis to
identify bottlenecks.
 collected information can be used with ksar to plot graphs.

Sar examples
CPU Usage of ALL CPUs (sar -u)
 This gives the cumulative real-time CPU usage of all CPUs.
 sar -u 1 3 Displays real time CPU usage every 1 second
for 3 times.
 “-P ALL” indicates that it should displays statistics for ALL
the individual Cores.
%user - % of cpu utilz. that occurred while executing at user level
%nice – at user level with nice priority.
%system – at sytem level
%iowait - % of time that cpu were idle during which s/m had an
outstanding disk I/O request
%idle – idle & s/m did not have an oustanding disk i/o
%steal - % of time spent in involuntary wait by cpu/cpu’s

Sar examples
• Memory Free and Used (sar -r)
 This reports the memory statistics.
 “1 3 reports for every 1 seconds a total of 3 times.″
 focus on “kbmemfree” and “kbmemused” for free and used memory.

Sar examples
• Overall I/O Activities (sar -b)
 This reports I/O statistics.
 tps – Transactions per second (this includes both read and write)
 rtps – Read transactions per second
 wtps – Write transactions per second
 bread/s – Bytes read per second
 bwrtn/s – Bytes written per second

Sar examples
• Individual Block Device I/O Activities (sar -d)
 To identify the activities by the individual block devices
(i.e a specific mount point, or LUN, or partition)
 The device name (DEV column) can display the actual
device name, use “sar -d”
tps –no of transfer per second
rd_sec/s – no sector read from device
wr_sec/s – no of sectors write from
device
avgrq-sz – avg size in sectors of the
req that were issued to device.
avgqu-sz – avg queue length of
request
await – avg time for i/o req.
svctm – avg service time for i/o req
%util – % of cputime during which i/o
req where issued

Sar examples
• Reports run queue and load average (sar -q)
 This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes.
runq-sz – run queue length
(no of task wtng for run time)
plist-sz – no of tasks in the task list

Sar examples
• To view previous records(sar –f)
 To view old day records
 Followed by path of old files
 Maintains one month records
sar -f /var/log/sa/sa15
sar -f /var/log/sa/sa15 -q 1 3
LC_ALL=C sar -f /var/log/sa/sa14 -A

iostat
 simple tool that will collect and show system
input and output storage device statistics.
 often used to trace storage device performance
issues including devices, local disks, remote disks
such as NFS.
%user - % of cpu utilz. that occurred while executing at user level
%nice – at user level with nice priority.
%system – at sytem level
%iowait - % of time that cpu were idle during which s/m had an
outstanding disk I/O request
%idle – idle & s/m did not have an oustanding disk i/o
%steal - % of time spent in involuntary wait by cpu/cpu’s
blk_read/s – amt of data read from device
blk_wrtn/s – amt of data write to device
Blk_read - total no of block reads
Blk_wrtn - total no of block writes

vmstat
 Summary information of Memory, Processes, Paging etc.
 used to display statistics of virtual memory, kernerl threads, disks, system processes, I/O
blocks, interrupts, CPU activity and much more.
r: The no of processes waiting for run time. b - The no of processes in uninterruptable sle
bi: Blocks sent to a block device (blocks/s). bo-: Blocks received from a block device (blocks/s).
si – Amount of memory swapped in from disk (kB/s). so – Amount of memory swapped to disk (kB/s).
in: The no of interrupts/s including the clock. cs : The number of context switches per second.
us: user time sy: system time id: idle time

pidstat
 pidstat is a part of sysstat utility.
 To monitor an individual process which is managed by kernel and generate report.
 Can also be used for monitoring the child processes of selected tasks.
 reports various statistics including CPU used by a process, disk usage statistics of a process,
statistics for threads associated with selected tasks and child processes.
Display statistics for all running process - pidstat -p ALL
Display statistics for a particular process - pidstat -p pid (process id of application)
Display statistics based on process name - pidstat -C “mysql”
Display Statistics of Dependent Process in
Tree Format using –t - pidstat -t -C "mysql"

mpstat
 mpstat is a part of sysstat utility.
 To monitor CPU utilization in our system.
 Will be more useful if our system has multiple processor.
Print CPU utilization per processors - mpstat -P 0 , mpstat -P 1
Print all CPU Utilization - mpstat -P ALL
Print CPU utilization using intervals - mpstat 3 4
Print mpstat version - mpstat -V

netstat
 monitoring incoming and outgoing network packets statistics as well as interface statistics.
 very useful tool for every system administrator to monitor network performance and
troubleshoot network related problems..

free
 built-in command that displays the total amount of free and used physical memory on your
machine.
 also displays the buffers used by the kernel at that given moment.

df
 df is an abbreviation for disk free.
 pre-installed program in all unix systems used to display the amount of available disk space for
filesystem which the user have access to.

du
 Linux “du” (Disk Usage) is a standard Unix/Linux command.
 used to check the information of disk usage of files and
directories on a machine.
 has many parameter options that can be used to get the
results in many formats.
 also displays the files and directory sizes in a recursively
manner.

ps
 basic commands for viewing the processes running on the system
 It provides a snapshot of the current processes along with detailed information like user id, cpu
usage, memory usage, command name etc.
 It does not display data in real time like top or htop commands.

lsof
 lsof meaning 'List Open Files' is used to find out which files are open by which process.
 when a disk cannot be un-mounted as it says the files are being used.

lsof some examples
 List User Specific Opened Files
lsof -u mydbops
 Exclude User with ‘^’ Character
lsof -i -u^root
 Kill all Activity of Particular User
kill -9 `lsof -t -u mydbops`
 List all Network Connections
lsof -i
 Search by PID
lsof -p 1

ping
 used to find out whether the peer host/gateway is reachable.
 how much time it takes for that data to be exchanged.
 default ping waits for 1 second before sending the next packet. increase or decrease using
option -i.
ping 122.166.212.214
ping -i 5 122.166.212.214
ping -c 3 122.166.212.214

telnet
 used for interactive communication with another host using the TELNET protocol.
 command mode.
Syntax : telnet host port

w & uptime
 w - displays information about the users currently on the machine, and their processes
 uptime - Tell how long the system has been running.

who & whoami
 who - prints information about all users who are currently logged in.
 Displays the username, line, and time of all currently logged-in sessions.
 whoami - This command prints the username associated with the current effective user ID.

history
 The history command can be used to list Bash's log of the commands you have typed:
 The history command performs one of several operations related to recently-executed
commands recorded in a history.
Syntax: history
Eg: history | less
 Search the history using Control+R

How MyDBOPS Can Help You
• MySQL Consulting
• MySQL Support
• Remote DBA
• 24 * 7 Support
Contact : mysqlsupport@mydbops.com

Linux monitoring and Troubleshooting for DBA's

More Related Content

What's hot (20)

Similar to Linux monitoring and Troubleshooting for DBA's (20)

More from Mydbops (20)

Recently uploaded (20)

Linux monitoring and Troubleshooting for DBA's