Some Oracle AWR observations

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
A Day of Real-World Performance
10/2/2018
Andrew Holdsworth, Tom Kyte, Graham Wood

Some Computer
Science Basics

Network Network Database
Server
Response Time v DB Time v Latency
Application
Server
End User
Total User Response Time
Time Line

Database Time – Total time spent in database
Db file sequential read
Run-queue
On CPU
User 1
Actual wait time
Recorded wait time
Db file sequential read
Run-queue Lock Wait
Latch
Wait
Run-queue
On CPU On CPU On CPUOn CPU On CPU On CPU On CPU On CPUOn CPU
User 2
Actual wait time Actual wait timeActual wait time
Recorded wait time Recorded wait time Recorded wait time
ON DEGRADED SYSTEM
Lock Wait
ON IDLE SYSTEM

Latency - Some Important Numbers
Block Location Access Time
L2 CPU cache ~ 1 nano sec ( 10-9 )
Virtual Memory ~ 1 micro sec ( 10-6 )
NUMA Far Memory ~ 10 micro sec ( 10-6 )
Flash Memory (PCI) ~ 0.01 milli sec ( 10-3 )
Flash Memory (Networked) ~ 0.1 milli sec ( 10-3 )
Disk I/O ~ 1-10 milli sec ( 10-3 )
Best Block Access Speeds

Database Performance Core Principles
• The Oracle database is a process based architecture and to perform
efficiently each process requires:
– To be efficiently scheduled by the O/S until the process completes the SQL statement,
or blocks on an operation required to complete the SQL statement e.g. Disk I/O
– If the process has to fight to get scheduled, or needs to be scheduled for an over
extended period of time due to SQL inefficiencies, or any blocking operation takes a
long time, then database performance will be poor
• Database performance engineers spend most of their time looking for CPU-
consuming processes and eliminating blocking events

Database Performance Core Principles
• To determine acceptable CPU utilization take a probabilistic approach to the
subject.
– If a CPU is 50% busy the chance of getting scheduled is 1 in 2
– If a CPU is 90% busy the chance of getting scheduled is 1 in10
• If the probabilities are used as indicator of the predictability of user response
time, then the variance in user response time becomes noticeable at about 60-
65%
• This has been observed in production and laboratory conditions for many years.

Impact of Too Many Processes
Database Core Principles
0
2000
4000
6000
8000
10000
12000
14000
16000
4 8 12 16 20 24 28 32
1 Proc/Core
10 Proc/Core Avg
50 Proc/Core Avg
10 Proc/Core Max
50 Proc/Core Max
10 Proc/Core Min
50 Proc/Core Min
#of CPUs
Tx/s

Connection Pooling

Connection Pools
Performance Data
The workload is increased by
doubling the load. System appears
scalable up to 60% CPU on the DB
server.

Connection Pools
Performance Data
A checkpoint is initiated, creating
a CPU spike that results in
unpredictable response time

Connection Pools
Performance Data
A slight increase to the
workload results in a
disproportionate CPU increase
and response time degrades.
System monitoring tools
become unreliable

Connection Pools
Performance Data
Reducing the connection pool by 50% results in
more application server queuing and less DB
processes in a wait state. No observable
improvement in response time or transaction rate
(value or consistency)

Connection Pools
Performance Data
Connection pool reduced to 96.
Note improvement in response time
and transaction rate.
CPU utilization is reduced.

Resource Management
Performance Data
By reducing the CPU_COUNT in the
resource manager, the database can be
throttled back. Note the increase in
response time and wait event resmgr:
cpu quantum

Leaking

Leaking
• Intermittent error: “ORA-01000: Maximum number of cursors exceeded”.
Application server fails and must be restarted
• The DBA has suggested that the init.ora parameter open_cursors be reset
to 30,000 to make the problem “go away for a while”.
• Symptoms of cursor leaking
Observations

Performance Data
Leaking
Error message:
ORA-01000 Maximum open
cursors exceeded
“SQL*Net break/reset
to client”

Cursor Data
Leaking
Cursor list with Count > 1
implies “leaked” cursors

Leaking
• After a period of time, the system performance begins to decline and then
degrades rapidly
• After rapid degradation, the application servers time out and the system is
unavailable
• The DBA claims the database is not the problem and simply needs more
connections
– The init.ora parameter processes is increased to 20,000
Observations

Leaking
• Due to coding errors on exception handling, the application leaks
connections in the connection pool making them programmatically
impossible to use
• This reduces the effective size of the connection pool
• The remaining connection are unable to keep up with the incoming
workload
• The rate of connection leakage is accelerated until there are no useable
connections left in the pool
Session Leaking

Leaking
• Potential indicators of session leaking:
– Frequent application server resets
– init.ora parameters process and sessions set very high
– Configuration of large and dynamic connection pools
– Large number of idle connections connected to the database
– Free memory on database server continually reduced
– Presence of idle connection kill scripts or middleware configured to kill idle sessions
Session Leaking

Leaking
• Without warning, the database appears to hang and the application servers
time out simultaneously
• The DBA sees that all connections are waiting on a single lock held by a
process that has not been active for a while.
• Each time the problem occurs, the DBA responds by running a script to kill
sessions held by long time lock holders and allowing the system to restart.
Observations

Leaking
• Lock leaking is usually a side effect of session leaking and the exception
handling code failing to execute a commit or rollback in the exception
handling process.
• A leaked session may be programmatically lost to the connection while
holding locks and uncommitted changes to the database.
Lock Leaking

Leaking
• Programming error impact:
– Potential system hangs: all connections queue up for the held lock
– Potential database logical corruptions: end users may have thought transactions were
committed when in fact they have not been
– If sessions return to the connection pool but still have uncommitted changes, it is not
deterministic, if and/or when the changes are committed or rolled back. This is a
serious data integrity issue.
Lock Leaking

Leaking
• Developer Bugs
– Incorrect/untested exception handling
• Cursor, session and lock leaking
– High values for init.ora ( open_cursors, processes, sessions )
– Idle process and lock holder kill scripts
– Oversized connection pools of largely idle processes
How to Develop High Performance Applications

Database / Middleware
Interaction

Scenario
Database / Middleware Interaction
• Devices ship files.
• Files read and
processed by
multiple
application
servers
• Each application
server uses
multiple threads
that connect to
database through
a connection pool
which is
distributed by a
scan listener over
two instances.

Problem
• It’s too slow
• It’s a problem
with the
database
– Look at all
those waits
• Need to be able
to process an
order of
magnitude
more data
• Obviously need
to move to
Hadoop

Analysis
• Only small
amount of data
being
processed.
• Both instances
essentially idle
with most
processes
waiting in RAC
and
concurrency
waits.

Solution
• Remove all of
those RAC waits
by running
against a single
database
instance.

Analysis
• Throughput up
by factor of 10x
• RAC waits gone
• CPU time
actually visible
• High
concurrency
waits
– Buffer busy
– Tx index
contention

Solution
• Reduce
contention
waits by
processing a file
entirely within a
single
application
server

Analysis
• Throughput
improved again
• Concurrency
events reduced
but still present

Solution
• Introduce
affinity for a
related set of
records to a
single thread by
hashing
• All records for
the same
primary key
processed by
single thread so
no contention in
index for same
primary key vale

Analysis
• More
throughput
• Log file sync
predominant
event
• CPU usage close
to core count

Solution
• Reintroduce
RAC to add
more CPU
resource
• Implement
separate service
for each
instance
• Connect
application
server to one
instance

Some Oracle AWR observations

More Related Content

What's hot (20)

Similar to Some Oracle AWR observations (20)

More from Connor McDonald (20)

Recently uploaded (20)

Some Oracle AWR observations