SlideShare a Scribd company logo

Ash masters : advanced ash analytics on Oracle

Download as PPT, PDF
Kyle Hailey, profile picture
Kyle Hailey

The document discusses database performance tuning. It recommends using Active Session History (ASH) and sampling sessions to identify the root causes of performance issues like buffer busy waits. ASH provides key details on sessions, SQL statements, wait events, and durations to understand top resource consumers. Counting rows in ASH approximates time spent and is important for analysis. Sampling sessions in real-time can provide the SQL, objects, and blocking sessions involved in issues like buffer busy waits.

Software
1 of 127
Downloaded 167 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Most read
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Most read
38
39
40
41
42
43
44
45
46
47
48
49
Most read
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
09/19/16 1 ASH Masters Kyle Hailey kylehailey.com Kylelf@gmail.com slideshare.net/khailey ASH SQL Query Repository
Save the Date COLLABORATE 17 registration will open on Thursday, October 27. Call for Speakers Submit your session presentation! The Call for Speakers is open until Friday, October 7 collaborate.ioug.org
How do you tune a Database? Database is running slow! ‱ first step? ‱ questions? ‱ tools? ‱ repeatable method?
What statistics do you look at? Disastrously difficult
Imagine Trying to Drive your Car And is updated once and hourAnd is updated once and hour Or would you like it toOr would you like it to look 
look 
 Would you want your dashboard to look like :Would you want your dashboard to look like :
Max CPU (yard stick) Top ActivityTop Activity SQLSQL SessionsSessions LOADLOAD Don’t always have OEM, but always have SQL
Ideas for Today ASHASH SamplingSampling WaitsWaits OEMOEM Where does the OEM data come from ?
Ideas for Today ASHASH SamplingSampling WaitsWaits OEMOEM
Copyright 2006 Kyle Hailey Statspack = always available ‱Install ‱Connect as SYSDBA ‱@?/rdbms/admin/spcreate.sql ‱Run ‱Exec statspack.snap; ‱Generate Reports ‱@?/rdbms/admin/spreport.sql AWR ‱ SQL> @?/rdbms/admin/awrrpt.sql Where do you get data ?
Sections (10g)Sections (10g) Instance description Host hardware Snapshot headline Cache information Load profile Instance efficiency Shared pool stats Top timed events Host CPU load Host / Instance CPU warning VM activity Memory usage Time model stats RAC statistics All wait events Background wait events Event histogram SQL x 9 Instance activity Log switches OS stats Latch parent and child Mutex Segment stats x 8 Dictionary cache Library cache RAC (GES) RAC (CR and CUR served) RAC (cache xfer x 2) RAC (Remastering) Streams x 7 Shared pool advisor Java pool advisor SGA resizing SGA target advisor SGA summary SGA detail SQL memory summary Resource limits Parameters Session details x 3 Tablespace I/O File I/O File I/O histogram Buffer pool Instance Recovery Buffer pool advisory Buffer busy waits Various PGA summaries PGA histogram PGA advisory PGA allocation summary PGA allocation top N Enqueue (lock) activity Undo stats x 2 Latch activity Latch miss details
Statspack MethodStatspack Method Load profile good for having a feel for the application and comparing two periods for changes Efficiency ratios misleading carry over from version 6 days Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time ---------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 ---------------------------------------- Summary Waits Who/When Big Picture
Latch FreeLatch Free Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time -------------------------------------------- ------------ ----------- -------- latch free 9,652 760 66.10 CPU time 248 21.62 PL/SQL lock timer 41 123 10.72 SQL*Net message from dblink 681 14 1.22 log file parallel write 128 1 .13 ------------------------------------------------------------- What Latch? There are 100s
Row Locks 10g+Row Locks 10g+ op 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time ----------------------------------------- ------------ ----------- ------ ------ enq: TX - row lock contention 59 160 2714 41.8 PL/SQL lock timer 4 117 29291 30.6 CPU time 28 7.2 buffer busy waits 1,217 18 15 4.7 log file parallel write 422 11 27 3.0 Who is waiting Who is blocking What is the SQL What is the row?
Buffer Busy WaitBuffer Busy Wait Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 ------------------------------------------------------------ Buffer Busy Wait – trying to modify a block Who blocks? What object? what is the SQL?
Statspack fails for analysis Solution : Sampling ASHASH SamplingSampling WaitsWaits OEMOEM
Example ‱ Look at Statspack: Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 ------------------------------------------------------------
What do we do? ‱buffer busy wait ?? ‱Need: ‱SQL ‱Sessions ‱Objects ‱Statspack or AWR fail
Solution V$session ‱Who is waiting (like “buffer busy waits” ) ‱While happening ‱Data ‱ Sql ‱ Session ‱ Object ‱ Type of buffer busy wait ‱ File and block involved in buffer busy wait ‱Problem: waits over, data is gone ‱Solution: Sample data
Sample Query select nvl(s.username,s.program) username, s.sid sid, s.serial# serial, s.sql_hash_value sql_hash_value SQLID , substr(decode(w.wait_time, 0, w.event, 'ON CPU'),1,15) event , w.p1 p1, w.p2 p2, w.p3 p3 from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER'; This query works since v7This query works since v7 SQLSQL StateState UserUser OnlyOnly ActiveActive UsersUsers DetailsDetails UniqueUnique IdentifiersIdentifiers
Sampling Output USERNAME SID SERIAL SQL_HASH_V EVENT P1 P2 P3 ---------- ----- -------- ---------- -------------------- -------- -------- ---- SYS 64 8717 4116021597 PL/SQL lock timer 300 0 0 SYS 58 19467 961168820 ON CPU 16508152 1 0 STARGUS 71 6251 1311875676 direct path write 201 2155902 127 (CJQ0) 9 1 0 rdbms ipc message 500 0 0 Run sample query every second and save into a table “v$ash” Create table v$ash as select 
 ; Insert into v$ash select 
 ; -- wait 1 sec Insert into v$ash select 
 ; -- wait 1 sec Insert into v$ash select 
 ; Insert into v$ash select 
 ;
Buffer busy wait type SQL> Select count(*), p3 from v$ash where event = 'buffer busy waits' group by p3; COUNT(*) P3 ---------- ---- 3423 1 Buffer Busy Wait P1 = file # P2 = block # P3 = block type From v$event_name or documentation
Buffer busy wait type Buffer Busy Wait P3 = block type Block types come from select rownum n, class from v$waitstat; N CLASS 1 data block 2 sort block 3 save undo block 4 segment header 5 save undo header 6 free list 7 extent map 8 1st level bmb 9 2nd level bmb 10 3rd level bmb 11 bitmap block 12 bitmap index block 13 file header block 14 unused 15 system undo header 16 system undo block 17 undo header
File and Block #s select count(*), p1 filen, p2 blockn from v$ash where event='buffer busy waits' group by p1, p2, hash_value; COUNT(*) FILEN BLOCKN ---------- -------- -------- 1 11 90644 2 11 90651 3 11 98233 1 11 104767 3 11 113291 1 11 119842 1 11 119856 3 11 121632 1 11 126334 Pick a File and Block to find object Buffer Busy Wait P1 = file # P2 = block # P3 = block type
Find Object column segment_name format a30 select owner, segment_name, segment_type, block_id, blocks+block_id from dba_extents where file_id = 11 and 126334 between block_id AND block_id + blocks-1; OWNER SEGMENT_NAME SEGMENT_TY BLOCK_ID BLOCKS+BLOCK_ID ---------- ------------------ ---------- ---------- --------------- SYSTEM TOTO1 TABLE 125201 127249 COUNT(*) FILEN BLOCKN ---------- -------- -------- 1 11 126334
What SQL ? SQL> select count(*), sql_hash_value 2 from v$ash 3 where event='buffer busy waits' 4 group by hash_value; COUNT(*) SQL_HASH_VALUE ---------- -------------- 3423 558666863 SQL_HASH_VALUE = 558666863 for every wait NOTE: SQL_ID better
SQL Statement ? select sql_text from v$sqltext where hash_value=558666863; SQL_TEXT ----------------------------------------------------- INSERT into toto1 values (:b1,lpad('a',1000,'a')) ‱ Insert statement ‱ Problem on a data block on ‱ table toto1 Solution: Freelists or ASSM
Lack of Free List S1 S2 S3 S4 4 Sessions running Insert into toto1 values (null, ‘a’); Commit; OBJN OTYPE FILEN BLOCKN SQL_ID BLOCK_TYPE ----------- ------ ------ ------ ------------- ------------ 54962 TOTO1 TABLE 16 45012 8gz51m9hg5yuf data block 54962 TOTO1 TABLE 16 161 8gz51m9hg5yuf segment header
Buffer Busy on Index Solutions 1. Hash Partitions 2. Reverse Keys
Sampling Summary ‱Sample v$session => find root causes ‱Same data is in Active Session History (ASH) v$active_session_history
Ideas for Today ASHASH SamplingSampling WaitsWaits OEMOEM
v$active_session_historyv$active_session_history SESSION_ID NUMBER SESSION_SERIAL# NUMBER USER_ID NUMBER SERVICE_HASH NUMBER SESSION_TYPE VARCHAR2(10) PROGRAM VARCHAR2(64) MODULE VARCHAR2(48) ACTION VARCHAR2(32) CLIENT_ID VARCHAR2(64) EVENT VARCHAR2(64) EVENT_ID NUMBER EVENT# NUMBER SEQ# NUMBER P1 NUMBER P2 NUMBER P3 NUMBER WAIT_TIME NUMBER TIME_WAITED NUMBER CURRENT_OBJ# NUMBER CURRENT_FILE# NUMBER CURRENT_BLOCK# NUMBER0 SQL_ID VARCHAR2(13) SQL_CHILD_NUMBER NUMBER SQL_PLAN_HASH_VALUE NUMBER SQL_OPCODE NUMBER QC_SESSION_ID NUMBER QC_INSTANCE_ID NUMBER SAMPLE_ID NUMBER SAMPLE_TIME TIMESTAMP(3) SESSION_STATE VARCHAR2(7) WAIT_TIME NUMBER TIME_WAITED NUMBER When Session SQL Wait State Duration
Primary Fields ofPrimary Fields of ASHASH SESSION_ID EVENT : what kind of wait SQL_ID SAMPLE_TIME Time Session SQL Wait SESSION_STATE : WAITING, ON CPU State When Who How What
Groupings – Top ConsumerGroupings – Top Consumer SESSION_ID SESSION_SERIAL# (identify SID reuse) SESSION_TYPE (FOREGROUND,BACKGROUND) CURRENT_OBJ# CURRENT_FILE# CURRENT_BLOCK# USER_ID (SYS, SYSTEM, SCOTT etc) SERVICE_HASH (OE,GL,HR) PROGRAM (SQL, JDBC, Forms etc) MODULE.ACTION (PLSQL tagging) CLIENT_ID (identifying users in session pool) SQL_ID QC_SESSION_ID (Query Coordinator) QC_INSTANCE_ID (RAC) EVENT + P1, P2, P3 Only for I/O, some locks and buffer busy waits
Counting is the key to ASH Seconds ~= count(*) => DB Time Select count(*) secs from ASH where {criteria} and {time period}; *** Key Point for ASH Queries ***
Counting is the key to ASH Select count(*)/{time period in seconds} AAS from ASH where {criteria} and {time period}; AAS = “Average Active Sessions” load on the database DB Time ~= count(*) AAS = DB Time/elapsed
AAS FormulasAAS Formulas Use CPU count as yardstick:  AAS < 1 Database is not blocked  AAS ~= 0 Database basically idle Problems are in the APP not DB  AAS < # of CPUs CPU available Are any single sessions 100% active? ‱ AAS > # of CPUs Could have performance problems  AAS >> # of CPUS There is a bottleneck Ideal world – one database solution track CPU at OS AAS > 1 still want to know if a single user is 100% active
When Developers sayWhen Developers say The Database is slowThe Database is slow
Ash masters : advanced ash analytics on Oracle
Ash masters : advanced ash analytics on Oracle
AAS ~= 0AAS ~= 0
Top CPU SessionTop CPU Session Top CPU consuming Session in last 5 minutes Select session_id, count(*) from v$active_session_history where session_state= ‘ON CPU‘ and SAMPLE_TIME > sysdate – (5/(24*60)) group by session_id order by count(*) desc; Select session_id, count(*) from v$active_session_history where session_state= ‘ON CPU‘ and SAMPLE_TIME > sysdate – (5/(24*60)) group by session_id order by count(*) desc; Who is the rogue session ?Who is the rogue session ?
Results Top CPU SessionResults Top CPU Session SESSION_ID COUNT(*) ---------- ---------- 257 299 263 62 256 32 264 9 277 3 SESSION_ID COUNT(*) ---------- ---------- 257 299 263 62 256 32 264 9 277 3 Not AAS, what doesNot AAS, what does count mean?count mean?
Top CPU SessionTop CPU Session Top CPU consuming Session in last 5 minutes Select session_id, count(*)/(5*60) – COUNT/ELAPSED=AAS 100*(count(*)/(5*60) ) -- % active from v$active_session_history where session_state= ‘ON CPU‘ and SAMPLE_TIME > sysdate – (5/(24*60)) group by session_id order by count(*) desc; Select session_id, count(*)/(5*60) – COUNT/ELAPSED=AAS 100*(count(*)/(5*60) ) -- % active from v$active_session_history where session_state= ‘ON CPU‘ and SAMPLE_TIME > sysdate – (5/(24*60)) group by session_id order by count(*) desc; Who is the rogue session ?Who is the rogue session ? Last 5 minutesLast 5 minutes
Results Top CPU SessionResults Top CPU Session SESSION_ID AAS %busy ---------- ---------- --------- 257 .99 99 263 .21 21 256 .11 11 264 .03 3 277 .01 1 SESSION_ID AAS %busy ---------- ---------- --------- 257 .99 99 263 .21 21 256 .11 11 264 .03 3 277 .01 1 AAS/100 = % active
Making a 0-100% bar Select session_id, count(*) , round((count(*)*100)/(5*60),0) "%", --- lpad('*', 10 * (count(*)/(5*60)), '*') "Bar” -- From v$active_session_history Where session_state= 'ON CPU' and SAMPLE_TIME > sysdate - (5/(24*60)) group by session_id order by count(*) desc / AAS per session 0-1 10 characters wide
CPU with BarsCPU with Bars SESSION_ID COUNT(*) % Bar ---------- ---------- ---------- ------------ 257 299 99 |**********| 263 62 21 |** | 256 32 11 |* | 264 9 3 | | 277 3 1 | | 258 1 0 | | 280 1 0 | | SESSION_ID COUNT(*) % Bar ---------- ---------- ---------- ------------ 257 299 99 |**********| 263 62 21 |** | 256 32 11 |* | 264 9 3 | | 277 3 1 | | 258 1 0 | | 280 1 0 | | Bar shows 10% incrementsBar shows 10% increments
Top Waiting SessionTop Waiting Session ‱ in last 5 minutes Select session_id, count(*) from v$active_session_history where session_state=‘WAITING’ and SAMPLE_TIME > SYSDATE - (5/(24*60)) group by session_id order by count(*) desc; Select session_id, count(*) from v$active_session_history where session_state=‘WAITING’ and SAMPLE_TIME > SYSDATE - (5/(24*60)) group by session_id order by count(*) desc;
Top Waiting Session ResultsTop Waiting Session Results SESSION_ID COUNT(*) ---------- ---------- 272 224 254 8 249 5 276 5 277 4 270 1 SESSION_ID COUNT(*) ---------- ---------- 272 224 254 8 249 5 276 5 277 4 270 1
Top SQL fromTop SQL from ASHASH select ash.SQL_ID , sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class, 'User I/O',1,0),0)) "WAIT" , sum(decode(ash.session_state,'WAITING', decode(en.wait_class, 'User I/O',1,0),0)) "IO" , sum(decode(ash.session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where SQL_ID is not NULL and en.event#=ash.event# group by sql_id order by sum(decode(session_state,'ON CPU',1,1)) desc select ash.SQL_ID , sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class, 'User I/O',1,0),0)) "WAIT" , sum(decode(ash.session_state,'WAITING', decode(en.wait_class, 'User I/O',1,0),0)) "IO" , sum(decode(ash.session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where SQL_ID is not NULL and en.event#=ash.event# group by sql_id order by sum(decode(session_state,'ON CPU',1,1)) desc No time window specifiedNo time window specified
Top SQL fromTop SQL from ASHASH ResultsResults SQL_ID CPU WAITING IO TOTAL ------------- ---------- ---------- ---------- ---------- 4c1xvq9ufwcjc 23386 0 0 23386 6wjw6rz5uvbp3 99 0 23 122 968dm8hr9qd03 97 0 22 119 938jp5gasmrah 90 0 25 115 cv8xnv81kf582 42 0 9 51 6p9bzu19v965k 21 0 0 21 5zu8pxnun66bu 15 0 0 15 db2jr13nup72v 9 0 0 9 7ks5gnj38hghv 8 0 0 8 SQL_ID CPU WAITING IO TOTAL ------------- ---------- ---------- ---------- ---------- 4c1xvq9ufwcjc 23386 0 0 23386 6wjw6rz5uvbp3 99 0 23 122 968dm8hr9qd03 97 0 22 119 938jp5gasmrah 90 0 25 115 cv8xnv81kf582 42 0 9 51 6p9bzu19v965k 21 0 0 21 5zu8pxnun66bu 15 0 0 15 db2jr13nup72v 9 0 0 9 7ks5gnj38hghv 8 0 0 8 0 0 0 0 0 0 0 0 0 0 23 22 25 9 0 0 0 0
Top SessionTop Session select ash.session_id, ash.session_serial#, ash.user_id, ash.program, sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "WAITING" , sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "IO" , sum(decode(session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where en.event# = ash.event# group by session_id,user_id,session_serial#,program order by sum(decode(session_state,'ON CPU',1,1)) select ash.session_id, ash.session_serial#, ash.user_id, ash.program, sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "WAITING" , sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "IO" , sum(decode(session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where en.event# = ash.event# group by session_id,user_id,session_serial#,program order by sum(decode(session_state,'ON CPU',1,1)) select ash.session_id, ash.session_serial#, ash.user_id, ash.program, sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "WAITING" , sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "IO" , sum(decode(session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where en.event# = ash.event# group by session_id,user_id,session_serial#,program order by sum(decode(session_state,'ON CPU',1,1)) No time window specifiedNo time window specified
Top Session ResultsTop Session Results SESSION_ID SERIAL# USER_ID PROGRAM CPU WAITING IO ---------- ------- ------- ------------------------- ----- ------- ---- 247 61970 1 sqlplus 11698 0 0 277 1 0 oracle@labsfrh903 (LGWR) 14 21 0 276 1 0 oracle@labsfrh903 (CKPT) 19 10 0 278 1 0 oracle@labsfrh903 (DBW0) 29 0 0 280 1 0 oracle@labsfrh903 (PMON) 19 0 0 254 22617 5 Executor.exe 13 0 3 SESSION_ID SERIAL# USER_ID PROGRAM CPU WAITING IO ---------- ------- ------- ------------------------- ----- ------- ---- 247 61970 1 sqlplus 11698 0 0 277 1 0 oracle@labsfrh903 (LGWR) 14 21 0 276 1 0 oracle@labsfrh903 (CKPT) 19 10 0 278 1 0 oracle@labsfrh903 (DBW0) 29 0 0 280 1 0 oracle@labsfrh903 (PMON) 19 0 0 254 22617 5 Executor.exe 13 0 3
Top Session w/ UsernameTop Session w/ Username select decode(nvl(to_char(s.sid),-1),-1,'DISCONNECTED','CONNECTED') "STATUS", topsession.session_id "SESSION_ID", u.name "NAME", topsession.program "PROGRAM", max(topsession.CPU) "CPU", max(topsession.WAITING) "WAITING", max(topsession.IO) "IO", max(topsession.TOTAL) "TOTAL" from {previous query} ) topsession , v$session s, user$ u where u.user# =topsession.user_id and /* outer join to v$session because the session might be disconnected */ topsession.session_id = s.sid (+) and topsession.session_serial# = s.serial# (+) group by topsession.session_id, topsession.session_serial#, topsession.user_id, topsession.program, s.username,s.sid,s.paddr,u.name order by max(topsession.TOTAL) desc select decode(nvl(to_char(s.sid),-1),-1,'DISCONNECTED','CONNECTED') "STATUS", topsession.session_id "SESSION_ID", u.name "NAME", topsession.program "PROGRAM", max(topsession.CPU) "CPU", max(topsession.WAITING) "WAITING", max(topsession.IO) "IO", max(topsession.TOTAL) "TOTAL" from {previous query} ) topsession , v$session s, user$ u where u.user# =topsession.user_id and /* outer join to v$session because the session might be disconnected */ topsession.session_id = s.sid (+) and topsession.session_serial# = s.serial# (+) group by topsession.session_id, topsession.session_serial#, topsession.user_id, topsession.program, s.username,s.sid,s.paddr,u.name order by max(topsession.TOTAL) desc No time window specifiedNo time window specified Connected?Connected? User nameUser name ASHASH
Top SessionTop Session Finding a Rogue UserFinding a Rogue User STATUS SESSION_ID NAME PROGRAM CPU WAITING IO ------------ ---------- ---------- ------------------------- ----- ------- ---- CONNECTED 247 CPU_Monger ChMgr304.exe 11704 0 0 CONNECTED 277 SYS oracle@labsfrh903 (LGWR) 14 19 0 CONNECTED 278 SYS oracle@labsfrh903 (DBW0) 29 0 0 CONNECTED 276 SYS oracle@labsfrh903 (CKPT) 18 9 0 CONNECTED 280 SYS oracle@labsfrh903 (PMON) 20 0 0 DISCONNECTED 255 SYSTEM Executor.exe 11 4 5 DISCONNECTED 257 SYSTEM Executor.exe 13 0 3 DISCONNECTED 255 SYSTEM Executor.exe 14 0 2 DISCONNECTED 257 SYSTEM Executor.exe 13 0 3 STATUS SESSION_ID NAME PROGRAM CPU WAITING IO ------------ ---------- ---------- ------------------------- ----- ------- ---- CONNECTED 247 CPU_Monger ChMgr304.exe 11704 0 0 CONNECTED 277 SYS oracle@labsfrh903 (LGWR) 14 19 0 CONNECTED 278 SYS oracle@labsfrh903 (DBW0) 29 0 0 CONNECTED 276 SYS oracle@labsfrh903 (CKPT) 18 9 0 CONNECTED 280 SYS oracle@labsfrh903 (PMON) 20 0 0 DISCONNECTED 255 SYSTEM Executor.exe 11 4 5 DISCONNECTED 257 SYSTEM Executor.exe 13 0 3 DISCONNECTED 255 SYSTEM Executor.exe 14 0 2 DISCONNECTED 257 SYSTEM Executor.exe 13 0 3
Queries can be complicated ‱Key to ASH ‱Difficulties ‱Pitfalls ‱Solution
Keys to ASH 1. Seconds = COUNT(*) 1. AAS = COUNT(*) / Elapsed 1. Combine CPU state with EVENT decode(session_state, 'ON CPU','ON CPU’, EVENT)
Queries can be complicated Difficulties ‱Query complicated ‱Time windows difficult ‱Filter can be tricky Pitfalls ‱time_waited misleading ‱WAIT_TIME is superfluous , if > 0 , means on CPU ‱dba_hist_active_sess_history only 1/10 samples kept Solution repository of pre-written queries
Solution: ASH Masters, Collection of ASH SQL scripts 1.Load Charts 2.Wait Analysis – locks, latches, bbw 3.SQL elapsed times 4.Top – sql, session, wait, procedure, object 5.I/O ‱Latency * - not from ASH, but important and related
59 Before ASH MastersBefore ASH Masters 1. ASHRPT ‱ Based entirely on v$active_session_history ‱ @?/rdbms/admin/ashrpt.sql ‱ Exec ASH_REPORT_TEXT/HTML select * from tableselect * from table (dbms_workload_repository.ash_report_text((dbms_workload_repository.ash_report_text( (select dbid from v$database),(select dbid from v$database), 1,1, sysdate – 1/24,sysdate – 1/24, sysdate )) ;sysdate )) ; remember ashrpt.sql
ASHRPTASHRPT ASH Report For TESTDB/testdbASH Report For TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC HostDB Name DB Id Instance Inst Num Release RAC Host ------------ ----------- ------------ -------- ----------- --- ------------------------ ----------- ------------ -------- ----------- --- ------------ TESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604aTESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604a CPUsCPUs SGA Size Buffer Cache Shared Pool ASH Buffer SizeSGA Size Buffer Cache Shared Pool ASH Buffer Size -------- ------------------ ------------------ ------------------ ------------------------------------ ------------------ ------------------ ------------------ 22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%)1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins)Elapsed Time: 5.0 (mins) Sample Count: 3,716Sample Count: 3,716 Average Active Sessions: 12.39Average Active Sessions: 12.39 Avg. Active Session per CPU: 6.19Avg. Active Session per CPU: 6.19 Report Target: None specifiedReport Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg ActiveAvg Active Event Event Class % Activity SessionsEvent Event Class % Activity Sessions ----------------------------------- --------------- ---------- --------------------------------------------- --------------- ---------- ---------- CPU + Wait for CPU CPU 67.98 8.42CPU + Wait for CPU CPU 67.98 8.42 enq: TX - row lock contention Application 23.98 2.97enq: TX - row lock contention Application 23.98 2.97 buffer busy waits Concurrency 4.66 0.58buffer busy waits Concurrency 4.66 0.58 latch: cache buffers chains Concurrency 2.26 0.28latch: cache buffers chains Concurrency 2.26 0.28 ASH Report For TESTDB/testdbASH Report For TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC HostDB Name DB Id Instance Inst Num Release RAC Host ------------ ----------- ------------ -------- ----------- --- ------------------------ ----------- ------------ -------- ----------- --- ------------ TESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604aTESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604a CPUsCPUs SGA Size Buffer Cache Shared Pool ASH Buffer SizeSGA Size Buffer Cache Shared Pool ASH Buffer Size -------- ------------------ ------------------ ------------------ ------------------------------------ ------------------ ------------------ ------------------ 22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%)1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins)Elapsed Time: 5.0 (mins) Sample Count: 3,716Sample Count: 3,716 Average Active Sessions: 12.39Average Active Sessions: 12.39 Avg. Active Session per CPU: 6.19Avg. Active Session per CPU: 6.19 Report Target: None specifiedReport Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg ActiveAvg Active Event Event Class % Activity SessionsEvent Event Class % Activity Sessions ----------------------------------- --------------- ---------- --------------------------------------------- --------------- ---------- ---------- CPU + Wait for CPU CPU 67.98 8.42CPU + Wait for CPU CPU 67.98 8.42 enq: TX - row lock contention Application 23.98 2.97enq: TX - row lock contention Application 23.98 2.97 buffer busy waits Concurrency 4.66 0.58buffer busy waits Concurrency 4.66 0.58 latch: cache buffers chains Concurrency 2.26 0.28latch: cache buffers chains Concurrency 2.26 0.28
ASH RPTASH RPT 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time ***
ASH Masters intro to github 1.Load Charts 2.Wait Analysis – locks, latches, bbw 3.SQL elapsed times 4.Top – sql, session, wait, procedure, object 5.I/O – size, object Extra: Latency * - not from ASH, but important
ASH Masters on github.com ‱https://github.com/khailey/ashmasters
ASH Masters README.md
1. Load Charts How to get a quick overview of all the data in the ASH ? Reproduce Active Session OEM tab in SQL 1. Load graph 2. Add Top waits 3. History
1. Load Charts:1. Load Charts: ash_graph.sqlash_graph.sql @ash_graph TIME AAS GRAPH ---------------- ------- ---------------------- 06-AUG 14:00:00 2.24 ++--------2--- 06-AUG 15:00:00 6.67 ++++------2---------- 06-AUG 16:00:00 2.59 ++--------2---- 06-AUG 17:00:00 1.26 ++----- 2 06-AUG 18:00:00 1.38 +++---- 2 06-AUG 19:00:00 1.74 ++------- 2 06-AUG 20:00:00 .99 +---- 2 06-AUG 21:00:00 1.22 ++----- 2 06-AUG 22:00:00 1.66 ++------ 2 06-AUG 23:00:00 1.08 +---- 2 07-AUG 00:00:00 .83 +--- 2 07-AUG 01:00:00 1.74 ++------- 2 07-AUG 02:00:00 2.47 ++--------2---- 07-AUG 03:00:00 6.59 +++-------2---------- @ash_graph TIME AAS GRAPH ---------------- ------- ---------------------- 06-AUG 14:00:00 2.24 ++--------2--- 06-AUG 15:00:00 6.67 ++++------2---------- 06-AUG 16:00:00 2.59 ++--------2---- 06-AUG 17:00:00 1.26 ++----- 2 06-AUG 18:00:00 1.38 +++---- 2 06-AUG 19:00:00 1.74 ++------- 2 06-AUG 20:00:00 .99 +---- 2 06-AUG 21:00:00 1.22 ++----- 2 06-AUG 22:00:00 1.66 ++------ 2 06-AUG 23:00:00 1.08 +---- 2 07-AUG 00:00:00 .83 +--- 2 07-AUG 01:00:00 1.74 ++------- 2 07-AUG 02:00:00 2.47 ++--------2---- 07-AUG 03:00:00 6.59 +++-------2---------- What is this ?
@ash_graph TIME AAS GRAPH ---------------- ------- ---------------------- 06-AUG 13:00:00 .33 +- 2 06-AUG 14:00:00 2.24 ++--------2--- 06-AUG 15:00:00 6.67 ++++------2---------- 06-AUG 16:00:00 2.59 ++--------2---- 06-AUG 17:00:00 1.26 ++----- 2 06-AUG 18:00:00 1.38 +++---- 2 06-AUG 19:00:00 1.74 ++------- 2 06-AUG 20:00:00 .99 +---- 2 06-AUG 21:00:00 1.22 ++----- 2 06-AUG 22:00:00 1.66 ++------ 2 06-AUG 23:00:00 1.08 +---- 2 07-AUG 00:00:00 .83 +--- 2 07-AUG 01:00:00 1.74 ++------- 2 07-AUG 02:00:00 2.47 ++--------2---- 07-AUG 03:00:00 6.59 +++-------2---------- 07-AUG 04:00:00 1.95 ++++++--- 2 07-AUG 05:00:00 3.08 +++++-----2------ @ash_graph TIME AAS GRAPH ---------------- ------- ---------------------- 06-AUG 13:00:00 .33 +- 2 06-AUG 14:00:00 2.24 ++--------2--- 06-AUG 15:00:00 6.67 ++++------2---------- 06-AUG 16:00:00 2.59 ++--------2---- 06-AUG 17:00:00 1.26 ++----- 2 06-AUG 18:00:00 1.38 +++---- 2 06-AUG 19:00:00 1.74 ++------- 2 06-AUG 20:00:00 .99 +---- 2 06-AUG 21:00:00 1.22 ++----- 2 06-AUG 22:00:00 1.66 ++------ 2 06-AUG 23:00:00 1.08 +---- 2 07-AUG 00:00:00 .83 +--- 2 07-AUG 01:00:00 1.74 ++------- 2 07-AUG 02:00:00 2.47 ++--------2---- 07-AUG 03:00:00 6.59 +++-------2---------- 07-AUG 04:00:00 1.95 ++++++--- 2 07-AUG 05:00:00 3.08 +++++-----2------ 1. Load Charts:1. Load Charts: ash_graph.sqlash_graph.sql
1. Load Charts:1. Load Charts: ash_graph.sqlash_graph.sql “-” = WAIT “+” = CPU which waits ?
1. Load Charts : ash_graph.sql START_TIME GRAPH -------------------- -------------------------- NOV 03 2013 22:02:00 ++---1------ NOV 03 2013 22:03:00 ++---1------ NOV 03 2013 22:04:00 +++--1--------------------- NOV 03 2013 22:05:00 +----1- NOV 03 2013 22:06:00 +++--1-- When waiting, which are the top waits ?
1. Load Charts:1. Load Charts: ash_graph_ash_graph_waitswaits.sql.sql Top Two Waits
1. Load Charts : ash_graph_waits.sql TO_CHAR( PCT1 FIRST PCT2 SECOND GRAPH -------- ---- --------------- ---- --------------- ---------------------------------------- 15 19:00 64 CPU 21 db file sequent ++o 4 15 20:00 63 CPU 19 read by other s ++++o- 4 15 21:00 31 db file sequent 24 CPU ++ooo---- 4 15 22:00 35 CPU 24 db file scatter +++++ooooooo--- 4 15 23:00 29 log file sync 25 db file sequent ++++ooooooooo-------4------------- 16 00:00 52 db file sequent 27 CPU ++++++++++oooooooooo4ooooooooooooooo-- 16 01:00 57 CPU 36 db file sequent +++++++++++oooooooo 4 16 02:00 38 db file sequent 21 CPU ++++++oooooooooooo--4--------- 16 03:00 69 db file sequent 20 CPU +++ooooooooooo 4 16 04:00 45 db file sequent 28 CPU o 4 16 05:00 58 db file sequent 24 CPU +ooo 4 16 06:00 41 db file sequent 39 CPU +oo 4 o I/O + cpu - wait OK, what about yesterday?
1. Load Charts History : dba_hist_active_sess_history ‱Week of ASH data !! (or more) ‱Only 1 in 10 ASH rows kept ‱Has DBID SELECT retention, snap_interval FROM wrm$_wr_control;SELECT retention, snap_interval FROM wrm$_wr_control; exec dbms_workload_repository.modify_snapshot_settings(14*24*60,60);exec dbms_workload_repository.modify_snapshot_settings(14*24*60,60); RETENTION SNAP_INTERVALRETENTION SNAP_INTERVAL ----------------- ---------------------------------- ----------------- +00014 00:00:00.0 +00000 01:00:00.0+00014 00:00:00.0 +00000 01:00:00.0 MinutesMinutesSave more history :
1. Load Charts: queries with history ‱ V$ACTIVE_SESSION_HISTORY - live ‱ ash_graph_waits.sql – minute buckets ‱ V$ACTIVE_SESSION_HISTORY + DBA_HIST_ACTIVE_SESS_HISTORY – combine ‱ ash_graph_waits_histash.sql – minute buckets ‱ DBA_HIST_ACTIVE_SESS_HISTORY - history only (DBID) ‱ ash_graph_histash_by_dbid.sql - input DBID ‱ ash_graph_histash_by_dbid_program.sql - input DBID and PROGRAM ‱ ash_graph_histash_by_dbid_sqlid.sql - input DBID and a SQL_ID Fast, in memory slower repositories
2. SQL Elapsed times (11g+ ) ASH 11g added sql_exec_id sql_exec_start SQL execs will have same above and ‱sql_id
2. SQL Elapsed times ‱ ash_sql_elapsed.sql longest running SQL ‱ ash_sql_elapsed_hist.sql with histogram of execution times ‱ ash_sql_elapsed_hist_longestid.sql exec id of longest query SQL_ID EXEC MX AV MIN ------------- ---------- ------- --------- ---------- 0fvrpk7476b7y 26 3068 133.1 0 1pjp66rxcj6tg 15 3106 767.7 57 SQL_ID EXECS MX MN AV 1 2 3 4 5 ------------- ---------- ---------- ---------- -------- ------ ------ ----- ----- ----- 5k7vccwjr5ahd 2653 1963 0 33.4 2623 15 8 4 3 ds8cz0fb8w147 161 2531 13 273.8 136 18 5 1 1 SQL_ID EXECS MX MN AV MAX_RUN_TIME LONGEST_SQ 1 2 3 4 5 ------------- ------ ---------- ---- -------- ----------------------------------- ---------- ------ ------ ----- ---- --- 2spgk3k0f7quz 251 29607 0 546.0 11-04-12 12:11:47 11-04-12 20:25:14 16781748 247 2 0 0 2 990m08w8xav7s 591 7681 0 52.0 11-04-13 00:39:27 11-04-13 02:47:28 16786685 587 0 0 2 2
2. SQL Elapsed times: ash_sql_elapsed.sql SQL_ID EXECS MX AV MIN ------------- ---------- ------- --------- ---------- 0fvrpk7476b7y 26 3068 133.1 0 1pjp66rxcj6tg 15 3106 767.7 57 8r5wuxk1dprhr 39 3510 841.0 24 0w5uu5kngyyty 21 3652 442.3 0 0hbv80w9ypy0n 161 4089 1183.9 0 71fwb4n6a92fv 49 4481 676.9 30 0bujgc94rg3fj 604 4929 24.7 0 64dqhdkkw63fd 1083 7147 7.2 0 What was the distribution?
2. SQL Elapsed times: ash_sql_elapsed_hist.sql SQL_ID Execs MX MN AV 1 2 3 4 5 ------------- ------ ---- --- ------ ---- ---- ---- --- --- 30hzp85f3qtxj 179 3029 29 111.0 178 0 0 0 1 Bnddu47dqmzqd 10 3138 68 732.3 7 1 1 0 1 8rua4c9agcqkb 978 820 0 72.7 830 115 21 9 2 Histogram buckets: elapsed time spread 820/5 = 164820 / 5 = 164 secs width of each bucket
2. SQL Elapsed times: ash_sql_elapsed_hist.sql SQL_ID Execs MX MN AV 1 2 3 4 5 ------------- ------ ---- --- ------ ---- ---- ---- --- --- 30hzp85f3qtxj 179 3029 29 111.0 178 0 0 0 1 Bnddu47dqmzqd 10 3138 68 732.3 7 1 1 0 1 8rua4c9agcqkb 978 820 0 72.7 830 115 21 9 2 820/5 = 164164 secs width of each bucket 1. 0 – 164 830executions 2. 165 – 328 115 executions 3. 329 – 492 21 executions 4. 493 – 656 9 executions 5. 657 – 820 2 executions
2. SQL Elapsed times: ash_sql_elapsed_hist.sql SQL_ID Execs MX MN AV 1 2 3 4 5 ------------- ------ ---- --- ------ ---- ---- ---- --- --- 30hzp85f3qtxj 179 3029 29 111.0 178 0 0 0 1 Bnddu47dqmzqd 10 3138 68 732.3 7 1 1 0 1 8rua4c9agcqkb 978 820 0 72.7 830 115 21 9 2 820/5 = 164 1. <= 164 830 executions 2. <= 328 115 executions 3. <= 492 21 executions 4. <= 656 9 executions 5. <= 820 2 executions
2. SQL Elapsed times: ash_sql_elapsed_hist.sql SQL_ID Execs MX MN AV 1 2 3 4 5 ------------- ------ ---- --- ------ ---- ---- ---- --- --- 30hzp85f3qtxj 179 3029 29 111.0 178 0 0 0 1 Bnddu47dqmzqd 10 3138 68 732.3 7 1 1 0 1 8rua4c9agcqkb 978 820 0 72.7 830 115 21 9 2 820/5 = 164 Outlier
2. SQL Elapsed Times ash_sql_elapsed_hist_longestid.sql SQL_ID Execs MX MN AV MAX_RUN_TIME LONGEST_SQ 1 2 3 4 5 ------------- ---- ------ ---- ------ ------------------------------------------- ---------- ---- ----- ---- ---- --- 30hzp85f3qtxj 179 3029 29 111.0 11-11-07 13:46:12 11-11-07 14:36:41 16777250 177 1 0 0 1 C61wk6d7ssxxc 20 1801 258 743.0 11-11-07 12:01:02 11-11-07 12:31:03 16777216 13 2 0 0 5 769uu28qm4thw 17 1427 0 479.0 11-11-07 14:04:02 11-11-07 14:27:49 16777762 10 2 2 1 2 Fuzcbdt08xjcd 74 315 33 248.0 11-11-07 12:10:05 11-11-07 12:15:20 16790567 5 4 9 5 51 Bkv51bug8ag7c 29 307 1 36.0 11-11-07 11:58:30 11-11-07 12:03:37 16790749 26 2 0 0 1 1wgsn7mmf6kqc 131 299 0 6.0 11-11-07 12:01:08 11-11-07 12:06:07 16784143 130 0 0 0 1 3b7q0hd1q8pw0 17 288 1 118.0 11-11-07 09:42:21 11-11-07 09:47:09 16794374 7 4 0 1 5 1v6yyfy630rkj 13 135 42 65.0 11-11-07 07:49:12 11-11-07 07:51:27 16816140 7 4 1 0 1 1t715k5p9uxxx 50 95 0 32.0 11-11-07 05:17:55 11-11-07 05:19:30 16777225 32 1 1 0 16 A98fbc69py0us 11 91 59 74.0 11-11-07 14:55:55 11-11-07 14:57:26 16777389 1 4 3 2 1 0g53kf4gr3vrg 25 53 32 46.0 11-11-07 15:35:31 11-11-07 15:36:24 16777588 1 0 5 14 5 OK found the longest but ‱When did it happen? ‱What was the sql_exec_id of the longest? start End
2. SQL Elapsed Times – lookup by SQL ID Select decode(session_state,'ON CPU','ON CPU’, event) WAIT From v$active_session_history Where sql_exec_id = 16777217 Order by sample_id WAIT ------------------------------------------------ db file parallel read db file sequential read ON CPU ON CPU ON CPU ON CPU
3. Wait Analysis: v$active_session_history ‱ Buffer busy waits ‱ Enqueue transaction waits
Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 ------------------------------------------------------------ Buffer Busy Wait – trying to modify a block Who blocks? What object? what is the SQL? AWR fails 3. Wait Analysis: buffer busy waits
3. Wait Analysis: buffer busy waits OBJN OTYPE FILEN BLOCKN SQL_ID BLOCK_TYPE ------------------------- ----------- ------ ------------- ------------ 53218 BBW_INDEX_VAL_I INDEX 1 64826 97dgthz60u28d data block 1 53218 BBW_INDEX_VAL_I INDEX 1 64826 gypmcfzruu249 data block 1 53218 BBW_INDEX_VAL_I INDEX 1 64826 2vd1w5kgnfa5n data block 1 53218 BBW_INDEX_VAL_I INDEX 1 64826 3p3qncvp2juxs data block 1 53218 BBW_INDEX_VAL_I INDEX 1 64826 6avm49ys4k7t6 data block 1 ash_bbw.sql To solve Buffer Busy Waits, need ‱Object ‱ Table w/ data block => ASM or Freelists, ‱ Index w/data block => reverse or hash partition ‱File ‱ Temp files problem with extents
3. Wait Analysis: Joining ASH with v$waitstat select o.object_name obj, o.object_type otype, ash.SQL_ID, w.class from v$active_session_history ash, ( select rownum class#, class from v$waitstat ) w, all_objects o where event='buffer busy waits' and w.class#(+)=ash.p3 and o.object_id (+)= ash.CURRENT_OBJ# Order by sample_time; OBJ OTYPE SQL_ID CLASS ------ ------ ------------- ------------------ TOTO1 TABLE 8gz51m9hg5yuf data block TOTO1 TABLE 8gz51m9hg5yuf data block TOTO1 TABLE 8gz51m9hg5yuf segment header TOTO1 TABLE 8gz51m9hg5yuf data block
3. Wait Analysis: How to get Class Name select rownum n,ws.class from v$waitstat; NAME P1 P2 P3 ----------------- ----- ------ ----- buffer busy waits file# block# class# NAME P1 P2 P3 ----------------- ----- ------ ----- buffer busy waits file# block# class# select * from v$event_name where name = 'buffer busy waits' N CLASS 1 data block 2 sort block 3 save undo block 4 segment header 5 save undo header 6 free list 7 extent map 8 1st level bmb 9 2nd level bmb 10 3rd level bmb 11 bitmap block 12 bitmap index block 13 file header block 14 unused 15 system undo header 16 system undo block P3 = class#, how do we get class name?
3. Wait Analysis: tx enqueues Top 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time ----------------------------------------- ------------ ----------- ------ ------ enq: TX - row lock contention 59 160 2714 41.8 PL/SQL lock timer 4 117 29291 30.6 CPU time 28 7.2 buffer busy waits 1,217 18 15 4.7 log file parallel write 422 11 27 3.0 Who is waiting? Who is blocking? What is the SQL? What is the row? Not in AWR report
3. Wait Analysis: buffer busy waits ‱ Waiter ‱ SESSION_ID ‱ SESSION_SERIAL# ‱ USER_ID ‱ Object ‱ CURRENT_OBJ# ‱ CURRENT_FILE# ‱ CURRENT_BLOCK# ‱ SQL Waiting - SQL_ID ‱ Blocker - BLOCKING_SESSION - BLOCKING_SESSION_STATUS - BLOCKING_SESSION_SERIAL# ‱ Lock Type and Mode - Event = Type (name) - P1 = Type | Mode v$active_session_historyv$active_session_history
3. Wait Analysis: tx enqueue waits select to_char(sample_time,’HH MI’) st, substr(event,0,20) lock_name, ash.session_id waiter, mod(ash.p1,16) lmode, ash.p2 p2, ash.p3 p3, o.object_name object, o.object_type otype, CURRENT_FILE# filen, CURRENT_BLOCK# blockn, ash.SQL_ID waiting_sql, BLOCKING_SESSION blocker from v$active_session_history ash, all_objects o where event like 'enq: T%' and o.object_id (+)= ash.CURRENT_OBJ# / Mode 6 row Mode 4 Other ash_enq.sql
3. Waits: enq TX 43. Waits: enq TX 4 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 Mode 4, foreign key Mode 4, unique key Mode 4, bitmap No Object File or Block
3. Waits: enq TX 43. Waits: enq TX 4 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 Mode 4, foreign key Mode 4, unique key Mode 4, bitmap Child Table Blocker SID=1
3. Waits: enq TX 43. Waits: enq TX 4 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 Mode 4, foreign key Mode 4, unique key Mode 4, bitmap Obect Index No File or Block
4. Top A.Procedure ‱ ash_top_procedure.sql A.Session ‱ ash_top_session.sql - wait, I/O and CPU time A.SQL i.ash_top_sql.sql - wait, I/O and CPU time ii.ash_top_sql_w_top_obj.sql - with top OBJ per SQL
10.2.0.3 ASH added10.2.0.3 ASH added PLSQL_ENTRY_OBJECT_ID PLSQL_ENTRY_SUBPROGRAM_ID PLSQL_OBJECT_ID PLSQL_SUBPROGRAM_ID 9509/19/16 ALL_PROCEDURES viewALL_PROCEDURES view object_id = plsql_object_idobject_id = plsql_object_id subprogram_id = plsql_subprogram_idsubprogram_id = plsql_subprogram_id 4A. Top: PROCEDURE ash_top_procedure.sql
4A. Top: PROCEDURE ash_top_procedure.sql COUNT(*) SQL_ID calling_code --------- ------------- ------------------------------------------ 2 1xxksrhwtz3zf ORDERENTRY.NEWORDER => DBMS_RANDOM.VALUE 2 1xxksrhwtz3zf ORDERENTRY.NEWORDER => DBMS_LOCK.SLEEP 13 1xxksrhwtz3zf ORDERENTRY.NEWORDER 16 0bzhqhhj9mpaa ORDERENTRY.NEWCUSTOMER 45 41zu158rqf4kf ORDERENTRY.BROWSEANDUPDATEORDERS
4B. Top: SESSION ash_top_session.sql STATUS SID NAME PROGRAM CPU WAITING IO TOTAL ------------ ----- ------------ ------------------------- ----- ------- ----- ------ DISCONNECTED 54 SYS oracle@source (J000) 1227 540 152 1919 DISCONNECTED 57 SYS oracle@source (J001) 725 160 18 903 DISCONNECTED 71 SYS sqlplus@source (TNS V1-V3 535 60 36 631 DISCONNECTED 67 SYSTEM LAB128.exe 187 182 148 517 CONNECTED 10 SYS oracle@source (DBW0) 267 171 0 438 CONNECTED 11 SYS oracle@source (LGWR) 10 357 0 367 DISCONNECTED 44 SYS sqlplus@source (TNS V1-V3 103 158 15 276 CONNECTED 53 SYSTEM JDBC Thin Client 129 33 0 162 CONNECTED 36 SYSMAN OMS 6 114 0 120
4Ci. Top: SQL ash_top_sql.sql SQL_ID PLAN_HASH TYPE CPU WAIT IO TOTAL ------------- ---------- ---------- ------- ----- ------- ------ fdtr6ds1nxm5r 1269593971 SELECT 9345 9 84277 93631 5aa7r665a07n6 3761618565 SELECT 82040 6 850 82896 2x8kgb0s9q1zh 0 PL/SQL EXE 75967 1575 1186 78728 4qraawku9303p 2256852101 SELECT 46995 45 3196 50236 3h7agx5ndadrf 1006614515 SELECT 2068 1 24914 26983 8p2cyq3gdgau5 2108428761 SELECT 22345 489 313 23147 7vpqr5zhjm08v 4089415459 SELECT 422 0 19655 20077 7sf51jrzq6y3c 485652470 INSERT 1727 1418 15471 18616 6cy7mc2kj0u4z 3041431021 SELECT 14272 0 0 14272
4Cii. Top: SQL ash_top_sql_w_top_obj.sql SQL_ID SQLPLANHASH AUD_ACT CPU WAIT IO TOTAL PCT_IO TOP_OBJ ------------- ----------- ------- ---- ----- ---- ------- ------- -------------- dc9fkz2t3b9p8 1909389838 SELECT 3685 0 5 3690 60 PS_F_ABS_EXPTN_CA 30hzp85f3qtxj 3802366046 SELECT 568 0 921 1489 100 PS_TAX_BALANCE c61wk6d7ssxxc 702141750 SELECT 117 6 1364 1487 98 PS_PAY_DEDUCTION c3y6kdm1uzkb7 2395607145 SELECT 39 0 1375 1414 87 PS_PAY_CHECK 6wsy8rpd0bw26 252818247 SELECT 1363 0 14 1377 64 PS_F_TIALAST_HIST
5. I/O – lots you can do I/O by ‱SQL ‱Object ‱File & Table Space ‱Parallel Query master and slaves ‱By type of I/O ‱ Db file sequential read ‱ Db file scattered read ‱ Direct path read ‱ Direct path read temp ‱ Direct path write ‱ Direct path write temp
5. I/O : iosql.sql I/O by SQL and top Objects for that sql AAS SQL_ID % OBJ TABLESPACE ----- ------------- --- --------------- ---------- .18 0yas01u2p9ch4 6 ITEM_PRODUCT_IX SOEINDEX 6 ORDER_ITEMS_UK SOEINDEX 88 ITEM_ORDER_IX SOEINDEX .32 6v6gm0fd1rgrz 6 WAIT_OBJECTS SYSTEM 94 UNDO UNDOTBS1
Extra : Latency 1. latency_eventmetric.sql - Now ‱ 60 seconds avg ‱ V$EVENTMETRIC 1. latency_system_event.sql - Historic ‱ Hour average ‱ DBA_HIST_SYSTEM_EVENT Can’t get latency from: avg(TIME_WAITED) !
latency: latency_eventmetric.sql NAME WAIT_COUNT AVGMS ------------------------- ---------- ---------- log file parallel write 12 3.213 log file sync 6 4.682 db file sequential read 0 0 db file scattered read 0 0 direct path read 0 0 direct path read temp 0 0 direct path write 0 0 direct path write temp 0 0 Last 60 seconds
latency: last 60 seconds select -- m.intsize_csec, n.name , round(m.time_waited,3) time_waited, m.wait_count, round(10*m.time_waited/nullif(m.wait_count,0),3) avgms from v$eventmetric m, v$event_name n where m.event_id=n.event_id and n.name in ( 'db file sequential read', 'db file scattered read', 'direct path read', 'direct path read temp', 'direct path write', 'direct path write temp', 'log file sync', 'log file parallel write' I/O Events v$event_metric only had event ids  Join to v$event_name latency_eventmetric.sql
Latency: each hour latency_system_event.sql BTIME AVG_MS ------------------------ ---------- 01-MAY-13 23:00 .199 02-MAY-13 00:00 .023 02-MAY-13 01:00 .031 02-MAY-13 02:00 .006 02-MAY-13 03:00 .017 02-MAY-13 04:00 .015 02-MAY-13 05:00 .013 02-MAY-13 06:00 .019 02-MAY-13 07:00 .017 DBA_HIST_SYSTEM_EVENT DBA_HIST_SNAPSHOT Input event name
Oramon.sh Connected, starting collect at Wed Dec 5 14:59:24 EST 2012 starting stats collecting single block logfile write multi block direct read direct read temp direct write temp ms IOP/s ms IOP/s ms IOP/s ms IOP/s ms IOP/s ms IOP/s 3.53 .72 16.06 .17 4.64 .00 115.37 3.73 .00 0 1.66 487.33 2.66 138.50 4.84 33.00 .00 .00 0 1.71 670.20 3.14 195.00 5.96 42.00 .00 .00 0 2.19 502.27 4.61 136.82 10.74 27.00 .00 .00 0 1.38 571.17 2.54 177.67 4.50 20.00 .00 .00 0 single block logfile write multi block direct read direct read temp direct write temp ms IOP/s ms IOP/s ms IOP/s ms IOP/s ms IOP/s ms IOP/s 3.22 526.36 4.79 135.55 .00 .00 .00 0 2.37 657.20 3.27 192.00 .00 .00 .00 0 1.32 591.17 2.46 187.83 .00 .00 .00 0 2.23 668.60 3.09 204.20 .00 .00 .00 .00 0 https://github.com/khailey/oramon Usage: oramon.sh [username] [password] [host] [sid] <port=1521> <runtime=3600>
Summary Load on database = AAS = DB Time/elapsed ~ = count(*)/elapsed Select count(*)/{time period in seconds} AAS from ASH where {criteria} and {time period}; Custom queries can give detailed info on ‱ ASH Graph ‱ SQL excution, max, min ‱ Top SQL, Session, Procedure, Object ‱ IO by SQL, Object, Tablespace ‱ Detailed wait analysis – BBW, Enqueues etc https://github.com/khailey/ashmasters
END
1. Load Charts: basics, fake query1. Load Charts: basics, fake query Select pseudo columns start_time , bucket_id , count(cpu) , count(waits) from v$active_session_history ash START_TIME BUCKET_ID CPU WAITS ---------- ------------- ---------- ---------- 25-NOV-13 55 397 28231 25-NOV-13 56 80 5631
1. Load Charts: basics, fake query1. Load Charts: basics, fake query Select pseudo columns start_time , bucket_id , count(cpu) , count(waits) from v$active_session_history ash START_TIME BUCKET_ID CPU WAITS ---------- ------------- ---------- ---------- 25-NOV-13 55 397 28231 25-NOV-13 56 80 5631
1. Load Charts: components1. Load Charts: components BUCKET ID ID = truncate ( seconds in date / seconds in bucket ) truncate (0/5) = 0 truncate (1/5) = 0 truncate (2/5) = 0 truncate (3/5) = 0 truncate (4/5) = 0 truncate (5/5) = 1 truncate (6/5) = 1 truncate (7/5) = 1 truncate (8/5) = 1 truncate (9/5) = 1 0- 5 secs = 0 6-10 secs = 1
1. Load Charts: components1. Load Charts: components BUCKET ID ID = truncate ( seconds in date / seconds in bucket ) (( Julian days * seconds in a day ) + (seconds today)) / bucket size in seconds
1. Load Charts: components1. Load Charts: components BUCKET ID ID = truncate ( seconds in date / seconds in bucket ) (( Julian days * seconds in a day ) + (seconds today)) / bucket size in seconds trunc((to_char(sample_time,'J')*(24*60*60)+ to_char(sample_time,'SSSSS'))/&v_secs)
1. Load Charts: components1. Load Charts: components BUCKET ID ID = truncate ( seconds in date / seconds in bucket ) (( Julian days * seconds in a day ) + (seconds today)) / bucket size in seconds trunc((to_char(sample_time,'J')*(24*60*60)+ to_char(sample_time,'SSSSS'))/&v_secs) Counts , sum(decode(session_state,'ON CPU',1,0)) cpu , sum(decode(session_state,'WAITING',1,0)) waits
1. Load Charts: real query1. Load Charts: real query select trunc((to_char(sample_time,'J')*(24*60*60)+ to_char(sample_time,'SSSSS'))/&v_secs) id , sum(decode(session_state,'ON CPU',1,0)) cpu , sum(decode(session_state,'WAITING',1,0)) waits from v$active_session_history ash -- dba_hist_active_sess_history group by trunc((to_char(sample_time,'J')*(24*60*60)+ to_char(sample_time,'SSSSS'))/&v_secs) ID CPU WAITS ---------- ---------- ---------- 4.2432E+10 1 4 4.2432E+10 9 18 4.2432E+10 8 25 4.2432E+10 9 26 Group by ID ID Waits CPU
1. Load Charts: how do you draw a load1. Load Charts: how do you draw a load line?line? rpad('+',round(( cpu/&v_secs)*&v_bars),'+’) || rpad('-',round((waits/&v_secs)*&v_bars),'-') || If the value of 1 AAS is 5 characters wide then +++++----- Above is AAS of 2 (1 wait, 1 CPU) AAS Width of AAS Def v_secs=60 -- size bucket in seconds Def v_bars=5 -- size of one AAS
1. Load Charts: basics : graphics1. Load Charts: basics : graphics rpad('+',round(( cpu/&v_secs)*&v_bars),'+’) || rpad('-',round((waits/&v_secs)*&v_bars),'-') || rpad(' ',p.value * &v_bars,' '), -- blanks +++++----- 1 +++++----- +++++1----- Trick, add in the number cores 1.Draw full line, cut at core # 2.Add core # 3.Draw full line again cut before first core # || || What about a yard stick ?
1. Load Charts: graphics1. Load Charts: graphics Def v_bars=5 -- size of one AAS in characters Def v_graph=80 substr( substr( rpad('+',round((cpu*&v_bars)/&v_secs),'+') || rpad('-',round((waits*&v_bars)/&v_secs),'-') || rpad(' ',p.value * &v_bars,' '), 0,(p.value * &v_bars)) || p.value || substr( rpad('+',round((cpu*&v_bars)/&v_secs),'+') || rpad('-',round((waits*&v_bars)/&v_secs),'-') || rpad(' ',p.value * &v_bars,' '), (p.value * &v_bars)) ,0,&v_graph) 2nd half of line Add core count First half of line +++++----- 1 +++++-----
1. Load Charts: basics1. Load Charts: basicsselect to_char(to_date( trunc((id*&v_secs)/ (24*60*60)) || ' ' || -- Julian days mod((id*&v_secs), (24*60*60)) -- seconds in the day , 'J SSSSS' ), 'MON DD YYYY HH24:MI:SS') start_time, substr( substr( rpad('+',round(( cpu/&v_secs)*&v_bars),'+’) || rpad('-',round((waits/&v_secs)*&v_bars),'-') || rpad(' ',p.value * &v_bars,' ’) ,0,(p.value * &v_bars)) || p.value || substr( rpad('+',round(( cpu/&v_secs)*&v_bars),'+’) || rpad('-',round((waits/&v_secs)*&v_bars),'-') || rpad(' ',p.value * &v_bars,' '), (p.value * &v_bars)) ,0,&v_graph) graph from ( select trunc((to_char(sample_time,'J')*(24*60*60)+to_char(sample_time,'SSSSS'))/&v_secs) id , sum(decode(session_state,'ON CPU',1,0)) cpu , sum(decode(session_state,'WAITING',1,0)) waits from v$active_session_history ash group by trunc((to_char(sample_time,'J')*(24*60*60)+to_char(sample_time,'SSSSS'))/&v_secs) ) aveact, v$parameter p where p.name='cpu_count' order by id / Bar Date (from ID) ID WAITS CPU RAW Graphics
1. Load Charts : events ash_graph_waits.sql BUCKET_ID EVENT EVENT_COUNT ---------- ------------------------------- ----------- 3537592817 control file heartbeat 2 3537592818 ADR block file read 1 3537592818 ARCH wait for process start 3 3 3537592818 ON CPU 9 3537592818 control file parallel write 4 Select trunc((to_char(sample_time,'J')*(24*60*60)+to_char(sample_time,'SSSSS'))/&v_secs) Bucket_id , decode(session_state,'ON CPU','ON CPU',ash.event) event , count(*) event_count From v$active_session_history ash group by trunc((to_char(sample_time,'J')*(24*60*60)+to_char(sample_time,'SSSSS'))/&v_secs) , decode(session_state,'ON CPU','ON CPU',ash.event)
1. Load Charts : per bucket, rank the waits ID EVENT RANK PCT CPU WAITS ---------- ----------------------------- ---- --- ---- ----- 3537592818 ON CPU 1 .290 9 0 3537592818 db file sequential read 2 .161 0 5 3537592818 log file switch (checkpoint i 3 .161 0 5 3537592818 control file parallel write 4 .129 0 4 3537592818 ARCH wait for process start 5 .096 0 3 3537592818 ADR block file read 6 .032 0 1 select id , event , row_number() over ( partition by id order by event_count desc ) rank , ratio_to_report( event_count ) over ( partition by id ) pct , sum(decode(event,'ON CPU',event_total,0)) cpu , sum(decode(event,'ON CPU',0,event_total)) waits from ( Previous Query ) chunks group by id, event, event_total Rank Rank %
1. Load Charts : raw data ash_graph_waits.sql select id , round(max(decode(top.rank,1,pct,null)),2) fpct , max( decode(top.rank,1,decode(top.event,'ON CPU','CPU',event),null)) first , round(max(decode(top.rank,2,pct,null)),2) spct , max( decode(top.rank,2,decode(top.event,'ON CPU','CPU',event),null)) second , sum(waits) waits , sum(cpu) cpu 
 ID FPCT FIRST SPCT SECOND WAITS CPU ---------- ----- --------------- ----- --------------- ---------- ------ 3537592817 1.00 control file he 2 0 3537592818 .29 CPU .16 db file sequent 22 9 3537592819 .72 control file pa .14 CPU 37 6 3537592820 .75 CPU .25 control file pa 1 3 3537592821 .91 CPU .09 direct path rea 1 10 3537592823 .75 CPU .25 db file paralle 1 3
2. SQL Elapsed times : raw data select sql_id, sql_exec_id, -- SAMPLE_TIME - START TIME (cast(sample_time as date) – cast(sql_exec_start as date)) * (60*60*24) secs from v$active_session_history where sql_exec_id is not null seconds query had been running = Sample_time – sql_exec_start now select max seconds for each sql_exec_id Seconds running id SQL_ID SQL_EXEC_ID SECS ------------- ----------- ------ acc988uzvjmmt 16777220 3 acc988uzvjmmt 16777220 2 acc988uzvjmmt 16777220 1 acc988uzvjmmt 16777220 0
2. SQL Elapsed times select sql_id, sql_exec_id, max(tm) tm from ( previous query ) Final execution time is max of all execs SQL_ID SQL_EXEC_ID TM ------------- ----------- ---------- acc988uzvjmmt 16777220 3 acc988uzvjmmt 16777335 15 acc988uzvjmmt 16779830 1 acc988uzvjmmt 16781502 7
2. SQL Elapsed times select sql_id, count(*), max(tm) mx, avg(tm) av, min(tm) min from ( previous query ) each SQL_ID ‱max ‱min ‱average execution times SQL_ID COUNT(*) MX AV MIN ------------- ---------- ------- --------- ---------- acc988uzvjmmt 4 15 6.5 1
2. SQL Elapsed times: ash_sql_elapsed.sql SQL_ID Executions MX AV MIN ------------- ---------- ------- --------- ---------- 0fvrpk7476b7y 26 3068 133.1 0 1pjp66rxcj6tg 15 3106 767.7 57 8r5wuxk1dprhr 39 3510 841.0 24 0w5uu5kngyyty 21 3652 442.3 0 0hbv80w9ypy0n 161 4089 1183.9 0 71fwb4n6a92fv 49 4481 676.9 30 0bujgc94rg3fj 604 4929 24.7 0 64dqhdkkw63fd 1083 7147 7.2 0 990m08w8xav7s 591 7681 51.8 0 2n5369dsuvn5a 16 10472 5726.8 303 2spgk3k0f7quz 251 29607 546.1 0 36pd759xym9tc 12 37934 23861.9 1391 497wh6n7hu14f 49 69438 5498.2 0
iosql.sql select sum(cnt) over ( partition by io.sql_id order by sql_id ) tcnt, io.sql_id, io.cnt cnt, io.aas aas, io.objn objn, io.obj obj, io.p1 p1, f.tablespace_name tablespace_name from ( select sql_id, count(*) cnt, round(count(*)/(&v_minutes*60),2) aas, CURRENT_OBJ# objn, nvl(o.object_name,decode(CURRENT_OBJ#,-1,0,CURRENT_OBJ#)) obj, o.object_type otype, ash.p1 from v$active_session_history ash ,all_objects o where ( event like 'db file s%' or event like 'direct%' ) and o.object_id (+)= ash.CURRENT_OBJ# and sample_time > sysdate - &v_minutes/(60*24) group by CURRENT_OBJ#,o.object_name ,o.object_type , ash.p1, sql_id ) io, dba_data_files f where f.file_id = io.p1 Order by tcnt, io.sql_id, io.cnt /

More Related Content

PPTX
AWR and ASH Deep Dive
Kellyn Pot'Vin-Gorman
 
PDF
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
John Beresniewicz
 
PDF
Ash architecture and advanced usage rmoug2014
John Beresniewicz
 
PDF
Tanel Poder - Scripts and Tools short
Tanel Poder
 
PDF
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Tanel Poder
 
PDF
Oracle db performance tuning
Simon Huang
 
PDF
Oracle Performance Tuning Fundamentals
Enkitec
 
PDF
Oracle Database SQL Tuning Concept
Chien Chung Shen
 
AWR and ASH Deep Dive
Kellyn Pot'Vin-Gorman
 
DB Time, Average Active Sessions, and ASH Math - Oracle performance fundamentals
John Beresniewicz
 
Ash architecture and advanced usage rmoug2014
John Beresniewicz
 
Tanel Poder - Scripts and Tools short
Tanel Poder
 
Troubleshooting Complex Oracle Performance Problems with Tanel Poder
Tanel Poder
 
Oracle db performance tuning
Simon Huang
 
Oracle Performance Tuning Fundamentals
Enkitec
 
Oracle Database SQL Tuning Concept
Chien Chung Shen
 

What's hot (20)

PDF
Chasing the optimizer
Mauro Pagano
 
PDF
Analyzing and Interpreting AWR
pasalapudi
 
PDF
Same plan different performance
Mauro Pagano
 
PPTX
Oracle sql high performance tuning
Guy Harrison
 
PDF
AWR Ambiguity: Performance reasoning when the numbers don't add up
John Beresniewicz
 
PDF
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder
 
PDF
Understanding oracle rac internals part 2 - slides
Mohamed Farouk
 
PPSX
Oracle Performance Tuning Fundamentals
Carlos Sierra
 
PPT
Performance Tuning With Oracle ASH and AWR. Part 1 How And What
udaymoogala
 
PDF
TFA Collector - what can one do with it
Sandesh Rao
 
PDF
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Aaron Shilo
 
PDF
Oracle Database performance tuning using oratop
Sandesh Rao
 
PDF
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAs
Zohar Elkayam
 
PDF
AWR & ASH Analysis
aioughydchapter
 
PPT
UKOUG, Oracle Transaction Locks
Kyle Hailey
 
PDF
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1
Tanel Poder
 
PDF
Oracle statistics by example
Mauro Pagano
 
PPTX
SQL Plan Directives explained
Mauro Pagano
 
PDF
Oracle Latch and Mutex Contention Troubleshooting
Tanel Poder
 
PPT
Using Statspack and AWR for Memory Monitoring and Tuning
Texas Memory Systems, and IBM Company
 
Chasing the optimizer
Mauro Pagano
 
Analyzing and Interpreting AWR
pasalapudi
 
Same plan different performance
Mauro Pagano
 
Oracle sql high performance tuning
Guy Harrison
 
AWR Ambiguity: Performance reasoning when the numbers don't add up
John Beresniewicz
 
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 2
Tanel Poder
 
Understanding oracle rac internals part 2 - slides
Mohamed Farouk
 
Oracle Performance Tuning Fundamentals
Carlos Sierra
 
Performance Tuning With Oracle ASH and AWR. Part 1 How And What
udaymoogala
 
TFA Collector - what can one do with it
Sandesh Rao
 
Exploring Oracle Database Performance Tuning Best Practices for DBAs and Deve...
Aaron Shilo
 
Oracle Database performance tuning using oratop
Sandesh Rao
 
Oracle Database Performance Tuning Advanced Features and Best Practices for DBAs
Zohar Elkayam
 
AWR & ASH Analysis
aioughydchapter
 
UKOUG, Oracle Transaction Locks
Kyle Hailey
 
Tanel Poder - Troubleshooting Complex Oracle Performance Issues - Part 1
Tanel Poder
 
Oracle statistics by example
Mauro Pagano
 
SQL Plan Directives explained
Mauro Pagano
 
Oracle Latch and Mutex Contention Troubleshooting
Tanel Poder
 
Using Statspack and AWR for Memory Monitoring and Tuning
Texas Memory Systems, and IBM Company
 
Ad

Viewers also liked (20)

PPT
Intro to ASH
Kyle Hailey
 
PDF
oracle cloud with 2 nodes processing
mahdi ahmadi
 
PPTX
Virtual Data : Eliminating the data constraint in Application Development
Kyle Hailey
 
PPT
Rmoug ashmaster
Kyle Hailey
 
PPSX
Sql Performance Tuning with ASH &amp; AWR: Real World Use Cases
vbarun01
 
PPT
Jonathan Lewis explains Delphix
Kyle Hailey
 
PPTX
Mark Farnam : Minimizing the Concurrency Footprint of Transactions
Kyle Hailey
 
PDF
Dan Norris: Exadata security
Kyle Hailey
 
PPTX
OakTable World Sep14 clonedb
Connor McDonald
 
PPTX
Indexes: Structure, Splits and Free Space Management Internals
Christian Antognini
 
PPTX
OakTable World 2015 - Using XMLType content with the Oracle In-Memory Column...
Marco Gralike
 
PDF
Profiling the logwriter and database writer
Kyle Hailey
 
PDF
Oaktable World 2014 Kevin Closson: SLOB – For More Than I/O!
Kyle Hailey
 
PDF
delphix-ebook-using-data-effectively-compliance-banking-1
Jes Breslaw
 
PDF
How to find and fix your Oracle application performance problem
Cary Millsap
 
PPT
OOUG - Oracle Performance Tuning with AAS
Kyle Hailey
 
PPT
Oracle Open World Thursday 230 ashmasters
Kyle Hailey
 
PPTX
Data control
Kyle Hailey
 
PDF
Oracle Database on ACFS: a perfect marriage?
Ludovico Caldara
 
PDF
Oaktable World 2014 Toon Koppelaars: database constraints polite excuse
Kyle Hailey
 
Intro to ASH
Kyle Hailey
 
oracle cloud with 2 nodes processing
mahdi ahmadi
 
Virtual Data : Eliminating the data constraint in Application Development
Kyle Hailey
 
Rmoug ashmaster
Kyle Hailey
 
Sql Performance Tuning with ASH &amp; AWR: Real World Use Cases
vbarun01
 
Jonathan Lewis explains Delphix
Kyle Hailey
 
Mark Farnam : Minimizing the Concurrency Footprint of Transactions
Kyle Hailey
 
Dan Norris: Exadata security
Kyle Hailey
 
OakTable World Sep14 clonedb
Connor McDonald
 
Indexes: Structure, Splits and Free Space Management Internals
Christian Antognini
 
OakTable World 2015 - Using XMLType content with the Oracle In-Memory Column...
Marco Gralike
 
Profiling the logwriter and database writer
Kyle Hailey
 
Oaktable World 2014 Kevin Closson: SLOB – For More Than I/O!
Kyle Hailey
 
delphix-ebook-using-data-effectively-compliance-banking-1
Jes Breslaw
 
How to find and fix your Oracle application performance problem
Cary Millsap
 
OOUG - Oracle Performance Tuning with AAS
Kyle Hailey
 
Oracle Open World Thursday 230 ashmasters
Kyle Hailey
 
Data control
Kyle Hailey
 
Oracle Database on ACFS: a perfect marriage?
Ludovico Caldara
 
Oaktable World 2014 Toon Koppelaars: database constraints polite excuse
Kyle Hailey
 
Ad

Similar to Ash masters : advanced ash analytics on Oracle (20)

PPT
Oracle 10g Performance: chapter 00 sampling
Kyle Hailey
 
PPT
Thomas+Niewel+ +Oracletuning
afa reg
 
PDF
Troubleshooting Complex Performance issues - Oracle SEG$ contention
Tanel Poder
 
PPT
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
John Kanagaraj
 
PPT
Using AWR for IO Subsystem Analysis
Texas Memory Systems, and IBM Company
 
PPT
AWR, ADDM, ASH, Metrics and Advisors.ppt
bugzbinny
 
PPT
Direct SGA access without SQL
Kyle Hailey
 
PPT
Kyle Hailey Oracle Performance IO Waits
cookie1969
 
PPT
11thingsabout11g 12659705398222 Phpapp01
Karam Abuataya
 
PPT
11 Things About11g
fcamachob
 
PPT
Oracle 10g Performance: chapter 00 statspack
Kyle Hailey
 
PPT
waits_events_and_the_geeks6043265(3).ppt
cookie1969
 
PPT
Wait Events 10g
sagai
 
PPTX
Oracle Basics and Architecture
Sidney Chen
 
PPT
Informix Warehouse Accelerator (IWA) features in version 12.1
Keshav Murthy
 
PDF
Performance Tuning Using oratop
Sandesh Rao
 
PPT
active_session_history_oracle_performance.ppt
cookie1969
 
PPT
Using AWR for SQL Analysis
Texas Memory Systems, and IBM Company
 
PPT
Less04 Instance
vivaankumar
 
PDF
New Tuning Features in Oracle 11g - How to make your database as boring as po...
Sage Computing Services
 
Oracle 10g Performance: chapter 00 sampling
Kyle Hailey
 
Thomas+Niewel+ +Oracletuning
afa reg
 
Troubleshooting Complex Performance issues - Oracle SEG$ contention
Tanel Poder
 
Your tuning arsenal: AWR, ADDM, ASH, Metrics and Advisors
John Kanagaraj
 
Using AWR for IO Subsystem Analysis
Texas Memory Systems, and IBM Company
 
AWR, ADDM, ASH, Metrics and Advisors.ppt
bugzbinny
 
Direct SGA access without SQL
Kyle Hailey
 
Kyle Hailey Oracle Performance IO Waits
cookie1969
 
11thingsabout11g 12659705398222 Phpapp01
Karam Abuataya
 
11 Things About11g
fcamachob
 
Oracle 10g Performance: chapter 00 statspack
Kyle Hailey
 
waits_events_and_the_geeks6043265(3).ppt
cookie1969
 
Wait Events 10g
sagai
 
Oracle Basics and Architecture
Sidney Chen
 
Informix Warehouse Accelerator (IWA) features in version 12.1
Keshav Murthy
 
Performance Tuning Using oratop
Sandesh Rao
 
active_session_history_oracle_performance.ppt
cookie1969
 
Using AWR for SQL Analysis
Texas Memory Systems, and IBM Company
 
Less04 Instance
vivaankumar
 
New Tuning Features in Oracle 11g - How to make your database as boring as po...
Sage Computing Services
 

More from Kyle Hailey (20)

PPTX
Hooks in postgresql by Guillaume Lelarge
Kyle Hailey
 
PPTX
Performance insights twitch
Kyle Hailey
 
PPT
History of database monitoring
Kyle Hailey
 
PPT
Successfully convince people with data visualization
Kyle Hailey
 
PPTX
DBTA Data Summit : Eliminating the data constraint in Application Development
Kyle Hailey
 
PPTX
Accelerate Develoment with VIrtual Data
Kyle Hailey
 
PPTX
Delphix and Pure Storage partner
Kyle Hailey
 
PDF
Martin Klier : Volkswagen for Oracle Guys
Kyle Hailey
 
PPTX
What is DevOps
Kyle Hailey
 
PPTX
Data as a Service
Kyle Hailey
 
PPTX
Data Virtualization: Revolutionizing data cloning
Kyle Hailey
 
PPTX
BGOUG "Agile Data: revolutionizing database cloning'
Kyle Hailey
 
PPTX
Denver devops : enabling DevOps with data virtualization
Kyle Hailey
 
PPTX
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Kyle Hailey
 
PPTX
DevOps, Databases and The Phoenix Project UGF4042 from OOW14
Kyle Hailey
 
PPT
Delphix for DBAs by Jonathan Lewis
Kyle Hailey
 
PPTX
Kscope 14 Presentation : Virtual Data Platform
Kyle Hailey
 
PPTX
Data Virtualization: revolutionizing database cloning
Kyle Hailey
 
PPTX
Delphix and DBmaestro
Kyle Hailey
 
PPTX
Agile Data: revolutionizing data and database cloning
Kyle Hailey
 
Hooks in postgresql by Guillaume Lelarge
Kyle Hailey
 
Performance insights twitch
Kyle Hailey
 
History of database monitoring
Kyle Hailey
 
Successfully convince people with data visualization
Kyle Hailey
 
DBTA Data Summit : Eliminating the data constraint in Application Development
Kyle Hailey
 
Accelerate Develoment with VIrtual Data
Kyle Hailey
 
Delphix and Pure Storage partner
Kyle Hailey
 
Martin Klier : Volkswagen for Oracle Guys
Kyle Hailey
 
What is DevOps
Kyle Hailey
 
Data as a Service
Kyle Hailey
 
Data Virtualization: Revolutionizing data cloning
Kyle Hailey
 
BGOUG "Agile Data: revolutionizing database cloning'
Kyle Hailey
 
Denver devops : enabling DevOps with data virtualization
Kyle Hailey
 
Oracle Open World 2014: Lies, Damned Lies, and I/O Statistics [ CON3671]
Kyle Hailey
 
DevOps, Databases and The Phoenix Project UGF4042 from OOW14
Kyle Hailey
 
Delphix for DBAs by Jonathan Lewis
Kyle Hailey
 
Kscope 14 Presentation : Virtual Data Platform
Kyle Hailey
 
Data Virtualization: revolutionizing database cloning
Kyle Hailey
 
Delphix and DBmaestro
Kyle Hailey
 
Agile Data: revolutionizing data and database cloning
Kyle Hailey
 

Recently uploaded (20)

PPTX
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
PPTX
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
PDF
Navsoft: AI-Powered Business Solutions & Custom Software Development
Navsoft
 
PDF
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
PDF
Download FL Studio Crack Latest version 2025 ?
ghrom2211g
 
DOCX
Greta — No-Code AI for Building Full-Stack Web & Mobile Apps
niloyislam1187
 
PDF
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
PPTX
Embracing Complexity in Serverless! GOTO Serverless Bengaluru
SheenBrisals
 
PDF
EN-Survey-Report-SAP-LeanIX-EA-Insights-2025.pdf
deepakkhaitan3
 
PPTX
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
PPTX
AMADEUS TRAVEL AGENT SOFTWARE | AMADEUS TICKETING SYSTEM
philipnathen82
 
PDF
How to Make Money in the Metaverse_ Top Strategies for Beginners.pdf
Herond Labs
 
PPTX
history of c programming in notes for students .pptx
Vijayakumari829030
 
PDF
How to Choose the Right IT Partner for Your Business in Malaysia
Asian Business Services & Technologies
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
PPTX
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
PDF
Complete Guide to Website Development in Malaysia for SMEs
Asian Business Services & Technologies
 
PDF
Product Update: Alluxio AI 3.7 Now with Sub-Millisecond Latency
Alluxio, Inc.
 
PDF
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
PPTX
Oracle Fusion HCM Cloud Demo for Beginners
DharanOracleerp
 
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
Navsoft: AI-Powered Business Solutions & Custom Software Development
Navsoft
 
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
Download FL Studio Crack Latest version 2025 ?
ghrom2211g
 
Greta — No-Code AI for Building Full-Stack Web & Mobile Apps
niloyislam1187
 
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
Embracing Complexity in Serverless! GOTO Serverless Bengaluru
SheenBrisals
 
EN-Survey-Report-SAP-LeanIX-EA-Insights-2025.pdf
deepakkhaitan3
 
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
AMADEUS TRAVEL AGENT SOFTWARE | AMADEUS TICKETING SYSTEM
philipnathen82
 
How to Make Money in the Metaverse_ Top Strategies for Beginners.pdf
Herond Labs
 
history of c programming in notes for students .pptx
Vijayakumari829030
 
How to Choose the Right IT Partner for Your Business in Malaysia
Asian Business Services & Technologies
 
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
Complete Guide to Website Development in Malaysia for SMEs
Asian Business Services & Technologies
 
Product Update: Alluxio AI 3.7 Now with Sub-Millisecond Latency
Alluxio, Inc.
 
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
Oracle Fusion HCM Cloud Demo for Beginners
DharanOracleerp
 

Ash masters : advanced ash analytics on Oracle

  • 2. Save the Date COLLABORATE 17 registration will open on Thursday, October 27. Call for Speakers Submit your session presentation! The Call for Speakers is open until Friday, October 7 collaborate.ioug.org
  • 3. How do you tune a Database? Database is running slow! ‱ first step? ‱ questions? ‱ tools? ‱ repeatable method?
  • 4. What statistics do you look at? Disastrously difficult
  • 5. Imagine Trying to Drive your Car And is updated once and hourAnd is updated once and hour Or would you like it toOr would you like it to look 
look 
 Would you want your dashboard to look like :Would you want your dashboard to look like :
  • 6. Max CPU (yard stick) Top ActivityTop Activity SQLSQL SessionsSessions LOADLOAD Don’t always have OEM, but always have SQL
  • 9. Copyright 2006 Kyle Hailey Statspack = always available ‱Install ‱Connect as SYSDBA ‱@?/rdbms/admin/spcreate.sql ‱Run ‱Exec statspack.snap; ‱Generate Reports ‱@?/rdbms/admin/spreport.sql AWR ‱ SQL> @?/rdbms/admin/awrrpt.sql Where do you get data ?
  • 10. Sections (10g)Sections (10g) Instance description Host hardware Snapshot headline Cache information Load profile Instance efficiency Shared pool stats Top timed events Host CPU load Host / Instance CPU warning VM activity Memory usage Time model stats RAC statistics All wait events Background wait events Event histogram SQL x 9 Instance activity Log switches OS stats Latch parent and child Mutex Segment stats x 8 Dictionary cache Library cache RAC (GES) RAC (CR and CUR served) RAC (cache xfer x 2) RAC (Remastering) Streams x 7 Shared pool advisor Java pool advisor SGA resizing SGA target advisor SGA summary SGA detail SQL memory summary Resource limits Parameters Session details x 3 Tablespace I/O File I/O File I/O histogram Buffer pool Instance Recovery Buffer pool advisory Buffer busy waits Various PGA summaries PGA histogram PGA advisory PGA allocation summary PGA allocation top N Enqueue (lock) activity Undo stats x 2 Latch activity Latch miss details
  • 11. Statspack MethodStatspack Method Load profile good for having a feel for the application and comparing two periods for changes Efficiency ratios misleading carry over from version 6 days Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time ---------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 ---------------------------------------- Summary Waits Who/When Big Picture
  • 12. Latch FreeLatch Free Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time -------------------------------------------- ------------ ----------- -------- latch free 9,652 760 66.10 CPU time 248 21.62 PL/SQL lock timer 41 123 10.72 SQL*Net message from dblink 681 14 1.22 log file parallel write 128 1 .13 ------------------------------------------------------------- What Latch? There are 100s
  • 13. Row Locks 10g+Row Locks 10g+ op 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time ----------------------------------------- ------------ ----------- ------ ------ enq: TX - row lock contention 59 160 2714 41.8 PL/SQL lock timer 4 117 29291 30.6 CPU time 28 7.2 buffer busy waits 1,217 18 15 4.7 log file parallel write 422 11 27 3.0 Who is waiting Who is blocking What is the SQL What is the row?
  • 14. Buffer Busy WaitBuffer Busy Wait Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 ------------------------------------------------------------ Buffer Busy Wait – trying to modify a block Who blocks? What object? what is the SQL?
  • 15. Statspack fails for analysis Solution : Sampling ASHASH SamplingSampling WaitsWaits OEMOEM
  • 16. Example ‱ Look at Statspack: Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 ------------------------------------------------------------
  • 17. What do we do? ‱buffer busy wait ?? ‱Need: ‱SQL ‱Sessions ‱Objects ‱Statspack or AWR fail
  • 18. Solution V$session ‱Who is waiting (like “buffer busy waits” ) ‱While happening ‱Data ‱ Sql ‱ Session ‱ Object ‱ Type of buffer busy wait ‱ File and block involved in buffer busy wait ‱Problem: waits over, data is gone ‱Solution: Sample data
  • 19. Sample Query select nvl(s.username,s.program) username, s.sid sid, s.serial# serial, s.sql_hash_value sql_hash_value SQLID , substr(decode(w.wait_time, 0, w.event, 'ON CPU'),1,15) event , w.p1 p1, w.p2 p2, w.p3 p3 from v$session s, v$session_wait w where w.sid=s.sid and s.status='ACTIVE' and s.type='USER'; This query works since v7This query works since v7 SQLSQL StateState UserUser OnlyOnly ActiveActive UsersUsers DetailsDetails UniqueUnique IdentifiersIdentifiers
  • 20. Sampling Output USERNAME SID SERIAL SQL_HASH_V EVENT P1 P2 P3 ---------- ----- -------- ---------- -------------------- -------- -------- ---- SYS 64 8717 4116021597 PL/SQL lock timer 300 0 0 SYS 58 19467 961168820 ON CPU 16508152 1 0 STARGUS 71 6251 1311875676 direct path write 201 2155902 127 (CJQ0) 9 1 0 rdbms ipc message 500 0 0 Run sample query every second and save into a table “v$ash” Create table v$ash as select 
 ; Insert into v$ash select 
 ; -- wait 1 sec Insert into v$ash select 
 ; -- wait 1 sec Insert into v$ash select 
 ; Insert into v$ash select 
 ;
  • 21. Buffer busy wait type SQL> Select count(*), p3 from v$ash where event = 'buffer busy waits' group by p3; COUNT(*) P3 ---------- ---- 3423 1 Buffer Busy Wait P1 = file # P2 = block # P3 = block type From v$event_name or documentation
  • 22. Buffer busy wait type Buffer Busy Wait P3 = block type Block types come from select rownum n, class from v$waitstat; N CLASS 1 data block 2 sort block 3 save undo block 4 segment header 5 save undo header 6 free list 7 extent map 8 1st level bmb 9 2nd level bmb 10 3rd level bmb 11 bitmap block 12 bitmap index block 13 file header block 14 unused 15 system undo header 16 system undo block 17 undo header
  • 23. File and Block #s select count(*), p1 filen, p2 blockn from v$ash where event='buffer busy waits' group by p1, p2, hash_value; COUNT(*) FILEN BLOCKN ---------- -------- -------- 1 11 90644 2 11 90651 3 11 98233 1 11 104767 3 11 113291 1 11 119842 1 11 119856 3 11 121632 1 11 126334 Pick a File and Block to find object Buffer Busy Wait P1 = file # P2 = block # P3 = block type
  • 24. Find Object column segment_name format a30 select owner, segment_name, segment_type, block_id, blocks+block_id from dba_extents where file_id = 11 and 126334 between block_id AND block_id + blocks-1; OWNER SEGMENT_NAME SEGMENT_TY BLOCK_ID BLOCKS+BLOCK_ID ---------- ------------------ ---------- ---------- --------------- SYSTEM TOTO1 TABLE 125201 127249 COUNT(*) FILEN BLOCKN ---------- -------- -------- 1 11 126334
  • 25. What SQL ? SQL> select count(*), sql_hash_value 2 from v$ash 3 where event='buffer busy waits' 4 group by hash_value; COUNT(*) SQL_HASH_VALUE ---------- -------------- 3423 558666863 SQL_HASH_VALUE = 558666863 for every wait NOTE: SQL_ID better
  • 26. SQL Statement ? select sql_text from v$sqltext where hash_value=558666863; SQL_TEXT ----------------------------------------------------- INSERT into toto1 values (:b1,lpad('a',1000,'a')) ‱ Insert statement ‱ Problem on a data block on ‱ table toto1 Solution: Freelists or ASSM
  • 27. Lack of Free List S1 S2 S3 S4 4 Sessions running Insert into toto1 values (null, ‘a’); Commit; OBJN OTYPE FILEN BLOCKN SQL_ID BLOCK_TYPE ----------- ------ ------ ------ ------------- ------------ 54962 TOTO1 TABLE 16 45012 8gz51m9hg5yuf data block 54962 TOTO1 TABLE 16 161 8gz51m9hg5yuf segment header
  • 28. Buffer Busy on Index Solutions 1. Hash Partitions 2. Reverse Keys
  • 29. Sampling Summary ‱Sample v$session => find root causes ‱Same data is in Active Session History (ASH) v$active_session_history
  • 31. v$active_session_historyv$active_session_history SESSION_ID NUMBER SESSION_SERIAL# NUMBER USER_ID NUMBER SERVICE_HASH NUMBER SESSION_TYPE VARCHAR2(10) PROGRAM VARCHAR2(64) MODULE VARCHAR2(48) ACTION VARCHAR2(32) CLIENT_ID VARCHAR2(64) EVENT VARCHAR2(64) EVENT_ID NUMBER EVENT# NUMBER SEQ# NUMBER P1 NUMBER P2 NUMBER P3 NUMBER WAIT_TIME NUMBER TIME_WAITED NUMBER CURRENT_OBJ# NUMBER CURRENT_FILE# NUMBER CURRENT_BLOCK# NUMBER0 SQL_ID VARCHAR2(13) SQL_CHILD_NUMBER NUMBER SQL_PLAN_HASH_VALUE NUMBER SQL_OPCODE NUMBER QC_SESSION_ID NUMBER QC_INSTANCE_ID NUMBER SAMPLE_ID NUMBER SAMPLE_TIME TIMESTAMP(3) SESSION_STATE VARCHAR2(7) WAIT_TIME NUMBER TIME_WAITED NUMBER When Session SQL Wait State Duration
  • 32. Primary Fields ofPrimary Fields of ASHASH SESSION_ID EVENT : what kind of wait SQL_ID SAMPLE_TIME Time Session SQL Wait SESSION_STATE : WAITING, ON CPU State When Who How What
  • 33. Groupings – Top ConsumerGroupings – Top Consumer SESSION_ID SESSION_SERIAL# (identify SID reuse) SESSION_TYPE (FOREGROUND,BACKGROUND) CURRENT_OBJ# CURRENT_FILE# CURRENT_BLOCK# USER_ID (SYS, SYSTEM, SCOTT etc) SERVICE_HASH (OE,GL,HR) PROGRAM (SQL, JDBC, Forms etc) MODULE.ACTION (PLSQL tagging) CLIENT_ID (identifying users in session pool) SQL_ID QC_SESSION_ID (Query Coordinator) QC_INSTANCE_ID (RAC) EVENT + P1, P2, P3 Only for I/O, some locks and buffer busy waits
  • 34. Counting is the key to ASH Seconds ~= count(*) => DB Time Select count(*) secs from ASH where {criteria} and {time period}; *** Key Point for ASH Queries ***
  • 35. Counting is the key to ASH Select count(*)/{time period in seconds} AAS from ASH where {criteria} and {time period}; AAS = “Average Active Sessions” load on the database DB Time ~= count(*) AAS = DB Time/elapsed
  • 36. AAS FormulasAAS Formulas Use CPU count as yardstick:  AAS < 1 Database is not blocked  AAS ~= 0 Database basically idle Problems are in the APP not DB  AAS < # of CPUs CPU available Are any single sessions 100% active? ‱ AAS > # of CPUs Could have performance problems  AAS >> # of CPUS There is a bottleneck Ideal world – one database solution track CPU at OS AAS > 1 still want to know if a single user is 100% active
  • 37. When Developers sayWhen Developers say The Database is slowThe Database is slow
  • 40. AAS ~= 0AAS ~= 0
  • 41. Top CPU SessionTop CPU Session Top CPU consuming Session in last 5 minutes Select session_id, count(*) from v$active_session_history where session_state= ‘ON CPU‘ and SAMPLE_TIME > sysdate – (5/(24*60)) group by session_id order by count(*) desc; Select session_id, count(*) from v$active_session_history where session_state= ‘ON CPU‘ and SAMPLE_TIME > sysdate – (5/(24*60)) group by session_id order by count(*) desc; Who is the rogue session ?Who is the rogue session ?
  • 42. Results Top CPU SessionResults Top CPU Session SESSION_ID COUNT(*) ---------- ---------- 257 299 263 62 256 32 264 9 277 3 SESSION_ID COUNT(*) ---------- ---------- 257 299 263 62 256 32 264 9 277 3 Not AAS, what doesNot AAS, what does count mean?count mean?
  • 43. Top CPU SessionTop CPU Session Top CPU consuming Session in last 5 minutes Select session_id, count(*)/(5*60) – COUNT/ELAPSED=AAS 100*(count(*)/(5*60) ) -- % active from v$active_session_history where session_state= ‘ON CPU‘ and SAMPLE_TIME > sysdate – (5/(24*60)) group by session_id order by count(*) desc; Select session_id, count(*)/(5*60) – COUNT/ELAPSED=AAS 100*(count(*)/(5*60) ) -- % active from v$active_session_history where session_state= ‘ON CPU‘ and SAMPLE_TIME > sysdate – (5/(24*60)) group by session_id order by count(*) desc; Who is the rogue session ?Who is the rogue session ? Last 5 minutesLast 5 minutes
  • 44. Results Top CPU SessionResults Top CPU Session SESSION_ID AAS %busy ---------- ---------- --------- 257 .99 99 263 .21 21 256 .11 11 264 .03 3 277 .01 1 SESSION_ID AAS %busy ---------- ---------- --------- 257 .99 99 263 .21 21 256 .11 11 264 .03 3 277 .01 1 AAS/100 = % active
  • 45. Making a 0-100% bar Select session_id, count(*) , round((count(*)*100)/(5*60),0) "%", --- lpad('*', 10 * (count(*)/(5*60)), '*') "Bar” -- From v$active_session_history Where session_state= 'ON CPU' and SAMPLE_TIME > sysdate - (5/(24*60)) group by session_id order by count(*) desc / AAS per session 0-1 10 characters wide
  • 46. CPU with BarsCPU with Bars SESSION_ID COUNT(*) % Bar ---------- ---------- ---------- ------------ 257 299 99 |**********| 263 62 21 |** | 256 32 11 |* | 264 9 3 | | 277 3 1 | | 258 1 0 | | 280 1 0 | | SESSION_ID COUNT(*) % Bar ---------- ---------- ---------- ------------ 257 299 99 |**********| 263 62 21 |** | 256 32 11 |* | 264 9 3 | | 277 3 1 | | 258 1 0 | | 280 1 0 | | Bar shows 10% incrementsBar shows 10% increments
  • 47. Top Waiting SessionTop Waiting Session ‱ in last 5 minutes Select session_id, count(*) from v$active_session_history where session_state=‘WAITING’ and SAMPLE_TIME > SYSDATE - (5/(24*60)) group by session_id order by count(*) desc; Select session_id, count(*) from v$active_session_history where session_state=‘WAITING’ and SAMPLE_TIME > SYSDATE - (5/(24*60)) group by session_id order by count(*) desc;
  • 48. Top Waiting Session ResultsTop Waiting Session Results SESSION_ID COUNT(*) ---------- ---------- 272 224 254 8 249 5 276 5 277 4 270 1 SESSION_ID COUNT(*) ---------- ---------- 272 224 254 8 249 5 276 5 277 4 270 1
  • 49. Top SQL fromTop SQL from ASHASH select ash.SQL_ID , sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class, 'User I/O',1,0),0)) "WAIT" , sum(decode(ash.session_state,'WAITING', decode(en.wait_class, 'User I/O',1,0),0)) "IO" , sum(decode(ash.session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where SQL_ID is not NULL and en.event#=ash.event# group by sql_id order by sum(decode(session_state,'ON CPU',1,1)) desc select ash.SQL_ID , sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class, 'User I/O',1,0),0)) "WAIT" , sum(decode(ash.session_state,'WAITING', decode(en.wait_class, 'User I/O',1,0),0)) "IO" , sum(decode(ash.session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where SQL_ID is not NULL and en.event#=ash.event# group by sql_id order by sum(decode(session_state,'ON CPU',1,1)) desc No time window specifiedNo time window specified
  • 50. Top SQL fromTop SQL from ASHASH ResultsResults SQL_ID CPU WAITING IO TOTAL ------------- ---------- ---------- ---------- ---------- 4c1xvq9ufwcjc 23386 0 0 23386 6wjw6rz5uvbp3 99 0 23 122 968dm8hr9qd03 97 0 22 119 938jp5gasmrah 90 0 25 115 cv8xnv81kf582 42 0 9 51 6p9bzu19v965k 21 0 0 21 5zu8pxnun66bu 15 0 0 15 db2jr13nup72v 9 0 0 9 7ks5gnj38hghv 8 0 0 8 SQL_ID CPU WAITING IO TOTAL ------------- ---------- ---------- ---------- ---------- 4c1xvq9ufwcjc 23386 0 0 23386 6wjw6rz5uvbp3 99 0 23 122 968dm8hr9qd03 97 0 22 119 938jp5gasmrah 90 0 25 115 cv8xnv81kf582 42 0 9 51 6p9bzu19v965k 21 0 0 21 5zu8pxnun66bu 15 0 0 15 db2jr13nup72v 9 0 0 9 7ks5gnj38hghv 8 0 0 8 0 0 0 0 0 0 0 0 0 0 23 22 25 9 0 0 0 0
  • 51. Top SessionTop Session select ash.session_id, ash.session_serial#, ash.user_id, ash.program, sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "WAITING" , sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "IO" , sum(decode(session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where en.event# = ash.event# group by session_id,user_id,session_serial#,program order by sum(decode(session_state,'ON CPU',1,1)) select ash.session_id, ash.session_serial#, ash.user_id, ash.program, sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "WAITING" , sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "IO" , sum(decode(session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where en.event# = ash.event# group by session_id,user_id,session_serial#,program order by sum(decode(session_state,'ON CPU',1,1)) select ash.session_id, ash.session_serial#, ash.user_id, ash.program, sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) - sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "WAITING" , sum(decode(ash.session_state,'WAITING', decode(en.wait_class,'User I/O',1, 0 ), 0)) "IO" , sum(decode(session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, v$event_name en where en.event# = ash.event# group by session_id,user_id,session_serial#,program order by sum(decode(session_state,'ON CPU',1,1)) No time window specifiedNo time window specified
  • 52. Top Session ResultsTop Session Results SESSION_ID SERIAL# USER_ID PROGRAM CPU WAITING IO ---------- ------- ------- ------------------------- ----- ------- ---- 247 61970 1 sqlplus 11698 0 0 277 1 0 oracle@labsfrh903 (LGWR) 14 21 0 276 1 0 oracle@labsfrh903 (CKPT) 19 10 0 278 1 0 oracle@labsfrh903 (DBW0) 29 0 0 280 1 0 oracle@labsfrh903 (PMON) 19 0 0 254 22617 5 Executor.exe 13 0 3 SESSION_ID SERIAL# USER_ID PROGRAM CPU WAITING IO ---------- ------- ------- ------------------------- ----- ------- ---- 247 61970 1 sqlplus 11698 0 0 277 1 0 oracle@labsfrh903 (LGWR) 14 21 0 276 1 0 oracle@labsfrh903 (CKPT) 19 10 0 278 1 0 oracle@labsfrh903 (DBW0) 29 0 0 280 1 0 oracle@labsfrh903 (PMON) 19 0 0 254 22617 5 Executor.exe 13 0 3
  • 53. Top Session w/ UsernameTop Session w/ Username select decode(nvl(to_char(s.sid),-1),-1,'DISCONNECTED','CONNECTED') "STATUS", topsession.session_id "SESSION_ID", u.name "NAME", topsession.program "PROGRAM", max(topsession.CPU) "CPU", max(topsession.WAITING) "WAITING", max(topsession.IO) "IO", max(topsession.TOTAL) "TOTAL" from {previous query} ) topsession , v$session s, user$ u where u.user# =topsession.user_id and /* outer join to v$session because the session might be disconnected */ topsession.session_id = s.sid (+) and topsession.session_serial# = s.serial# (+) group by topsession.session_id, topsession.session_serial#, topsession.user_id, topsession.program, s.username,s.sid,s.paddr,u.name order by max(topsession.TOTAL) desc select decode(nvl(to_char(s.sid),-1),-1,'DISCONNECTED','CONNECTED') "STATUS", topsession.session_id "SESSION_ID", u.name "NAME", topsession.program "PROGRAM", max(topsession.CPU) "CPU", max(topsession.WAITING) "WAITING", max(topsession.IO) "IO", max(topsession.TOTAL) "TOTAL" from {previous query} ) topsession , v$session s, user$ u where u.user# =topsession.user_id and /* outer join to v$session because the session might be disconnected */ topsession.session_id = s.sid (+) and topsession.session_serial# = s.serial# (+) group by topsession.session_id, topsession.session_serial#, topsession.user_id, topsession.program, s.username,s.sid,s.paddr,u.name order by max(topsession.TOTAL) desc No time window specifiedNo time window specified Connected?Connected? User nameUser name ASHASH
  • 54. Top SessionTop Session Finding a Rogue UserFinding a Rogue User STATUS SESSION_ID NAME PROGRAM CPU WAITING IO ------------ ---------- ---------- ------------------------- ----- ------- ---- CONNECTED 247 CPU_Monger ChMgr304.exe 11704 0 0 CONNECTED 277 SYS oracle@labsfrh903 (LGWR) 14 19 0 CONNECTED 278 SYS oracle@labsfrh903 (DBW0) 29 0 0 CONNECTED 276 SYS oracle@labsfrh903 (CKPT) 18 9 0 CONNECTED 280 SYS oracle@labsfrh903 (PMON) 20 0 0 DISCONNECTED 255 SYSTEM Executor.exe 11 4 5 DISCONNECTED 257 SYSTEM Executor.exe 13 0 3 DISCONNECTED 255 SYSTEM Executor.exe 14 0 2 DISCONNECTED 257 SYSTEM Executor.exe 13 0 3 STATUS SESSION_ID NAME PROGRAM CPU WAITING IO ------------ ---------- ---------- ------------------------- ----- ------- ---- CONNECTED 247 CPU_Monger ChMgr304.exe 11704 0 0 CONNECTED 277 SYS oracle@labsfrh903 (LGWR) 14 19 0 CONNECTED 278 SYS oracle@labsfrh903 (DBW0) 29 0 0 CONNECTED 276 SYS oracle@labsfrh903 (CKPT) 18 9 0 CONNECTED 280 SYS oracle@labsfrh903 (PMON) 20 0 0 DISCONNECTED 255 SYSTEM Executor.exe 11 4 5 DISCONNECTED 257 SYSTEM Executor.exe 13 0 3 DISCONNECTED 255 SYSTEM Executor.exe 14 0 2 DISCONNECTED 257 SYSTEM Executor.exe 13 0 3
  • 55. Queries can be complicated ‱Key to ASH ‱Difficulties ‱Pitfalls ‱Solution
  • 56. Keys to ASH 1. Seconds = COUNT(*) 1. AAS = COUNT(*) / Elapsed 1. Combine CPU state with EVENT decode(session_state, 'ON CPU','ON CPU’, EVENT)
  • 57. Queries can be complicated Difficulties ‱Query complicated ‱Time windows difficult ‱Filter can be tricky Pitfalls ‱time_waited misleading ‱WAIT_TIME is superfluous , if > 0 , means on CPU ‱dba_hist_active_sess_history only 1/10 samples kept Solution repository of pre-written queries
  • 58. Solution: ASH Masters, Collection of ASH SQL scripts 1.Load Charts 2.Wait Analysis – locks, latches, bbw 3.SQL elapsed times 4.Top – sql, session, wait, procedure, object 5.I/O ‱Latency * - not from ASH, but important and related
  • 59. 59 Before ASH MastersBefore ASH Masters 1. ASHRPT ‱ Based entirely on v$active_session_history ‱ @?/rdbms/admin/ashrpt.sql ‱ Exec ASH_REPORT_TEXT/HTML select * from tableselect * from table (dbms_workload_repository.ash_report_text((dbms_workload_repository.ash_report_text( (select dbid from v$database),(select dbid from v$database), 1,1, sysdate – 1/24,sysdate – 1/24, sysdate )) ;sysdate )) ; remember ashrpt.sql
  • 60. ASHRPTASHRPT ASH Report For TESTDB/testdbASH Report For TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC HostDB Name DB Id Instance Inst Num Release RAC Host ------------ ----------- ------------ -------- ----------- --- ------------------------ ----------- ------------ -------- ----------- --- ------------ TESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604aTESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604a CPUsCPUs SGA Size Buffer Cache Shared Pool ASH Buffer SizeSGA Size Buffer Cache Shared Pool ASH Buffer Size -------- ------------------ ------------------ ------------------ ------------------------------------ ------------------ ------------------ ------------------ 22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%)1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins)Elapsed Time: 5.0 (mins) Sample Count: 3,716Sample Count: 3,716 Average Active Sessions: 12.39Average Active Sessions: 12.39 Avg. Active Session per CPU: 6.19Avg. Active Session per CPU: 6.19 Report Target: None specifiedReport Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg ActiveAvg Active Event Event Class % Activity SessionsEvent Event Class % Activity Sessions ----------------------------------- --------------- ---------- --------------------------------------------- --------------- ---------- ---------- CPU + Wait for CPU CPU 67.98 8.42CPU + Wait for CPU CPU 67.98 8.42 enq: TX - row lock contention Application 23.98 2.97enq: TX - row lock contention Application 23.98 2.97 buffer busy waits Concurrency 4.66 0.58buffer busy waits Concurrency 4.66 0.58 latch: cache buffers chains Concurrency 2.26 0.28latch: cache buffers chains Concurrency 2.26 0.28 ASH Report For TESTDB/testdbASH Report For TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC HostDB Name DB Id Instance Inst Num Release RAC Host ------------ ----------- ------------ -------- ----------- --- ------------------------ ----------- ------------ -------- ----------- --- ------------ TESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604aTESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604a CPUsCPUs SGA Size Buffer Cache Shared Pool ASH Buffer SizeSGA Size Buffer Cache Shared Pool ASH Buffer Size -------- ------------------ ------------------ ------------------ ------------------------------------ ------------------ ------------------ ------------------ 22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%)1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01Analysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01Analysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins)Elapsed Time: 5.0 (mins) Sample Count: 3,716Sample Count: 3,716 Average Active Sessions: 12.39Average Active Sessions: 12.39 Avg. Active Session per CPU: 6.19Avg. Active Session per CPU: 6.19 Report Target: None specifiedReport Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg ActiveAvg Active Event Event Class % Activity SessionsEvent Event Class % Activity Sessions ----------------------------------- --------------- ---------- --------------------------------------------- --------------- ---------- ---------- CPU + Wait for CPU CPU 67.98 8.42CPU + Wait for CPU CPU 67.98 8.42 enq: TX - row lock contention Application 23.98 2.97enq: TX - row lock contention Application 23.98 2.97 buffer busy waits Concurrency 4.66 0.58buffer busy waits Concurrency 4.66 0.58 latch: cache buffers chains Concurrency 2.26 0.28latch: cache buffers chains Concurrency 2.26 0.28
  • 61. ASH RPTASH RPT 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 1) General info 2) Top User Events *** 3) Top Background Events 4) Top Event P1/P2/P3 Values 5) Top Service/Module 6) Top Client IDs 7) Top SQL Command Types 8) Top SQL Statements *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time *** 9) Top SQL using literals 10) Top Sessions *** 11) Top Blocking Sessions 12) Top Sessions running PQs 13) Top DB Objects 14) Top DB Files 15) Top Latches 16) Activity Over Time ***
  • 62. ASH Masters intro to github 1.Load Charts 2.Wait Analysis – locks, latches, bbw 3.SQL elapsed times 4.Top – sql, session, wait, procedure, object 5.I/O – size, object Extra: Latency * - not from ASH, but important
  • 63. ASH Masters on github.com ‱https://github.com/khailey/ashmasters
  • 65. 1. Load Charts How to get a quick overview of all the data in the ASH ? Reproduce Active Session OEM tab in SQL 1. Load graph 2. Add Top waits 3. History
  • 66. 1. Load Charts:1. Load Charts: ash_graph.sqlash_graph.sql @ash_graph TIME AAS GRAPH ---------------- ------- ---------------------- 06-AUG 14:00:00 2.24 ++--------2--- 06-AUG 15:00:00 6.67 ++++------2---------- 06-AUG 16:00:00 2.59 ++--------2---- 06-AUG 17:00:00 1.26 ++----- 2 06-AUG 18:00:00 1.38 +++---- 2 06-AUG 19:00:00 1.74 ++------- 2 06-AUG 20:00:00 .99 +---- 2 06-AUG 21:00:00 1.22 ++----- 2 06-AUG 22:00:00 1.66 ++------ 2 06-AUG 23:00:00 1.08 +---- 2 07-AUG 00:00:00 .83 +--- 2 07-AUG 01:00:00 1.74 ++------- 2 07-AUG 02:00:00 2.47 ++--------2---- 07-AUG 03:00:00 6.59 +++-------2---------- @ash_graph TIME AAS GRAPH ---------------- ------- ---------------------- 06-AUG 14:00:00 2.24 ++--------2--- 06-AUG 15:00:00 6.67 ++++------2---------- 06-AUG 16:00:00 2.59 ++--------2---- 06-AUG 17:00:00 1.26 ++----- 2 06-AUG 18:00:00 1.38 +++---- 2 06-AUG 19:00:00 1.74 ++------- 2 06-AUG 20:00:00 .99 +---- 2 06-AUG 21:00:00 1.22 ++----- 2 06-AUG 22:00:00 1.66 ++------ 2 06-AUG 23:00:00 1.08 +---- 2 07-AUG 00:00:00 .83 +--- 2 07-AUG 01:00:00 1.74 ++------- 2 07-AUG 02:00:00 2.47 ++--------2---- 07-AUG 03:00:00 6.59 +++-------2---------- What is this ?
  • 67. @ash_graph TIME AAS GRAPH ---------------- ------- ---------------------- 06-AUG 13:00:00 .33 +- 2 06-AUG 14:00:00 2.24 ++--------2--- 06-AUG 15:00:00 6.67 ++++------2---------- 06-AUG 16:00:00 2.59 ++--------2---- 06-AUG 17:00:00 1.26 ++----- 2 06-AUG 18:00:00 1.38 +++---- 2 06-AUG 19:00:00 1.74 ++------- 2 06-AUG 20:00:00 .99 +---- 2 06-AUG 21:00:00 1.22 ++----- 2 06-AUG 22:00:00 1.66 ++------ 2 06-AUG 23:00:00 1.08 +---- 2 07-AUG 00:00:00 .83 +--- 2 07-AUG 01:00:00 1.74 ++------- 2 07-AUG 02:00:00 2.47 ++--------2---- 07-AUG 03:00:00 6.59 +++-------2---------- 07-AUG 04:00:00 1.95 ++++++--- 2 07-AUG 05:00:00 3.08 +++++-----2------ @ash_graph TIME AAS GRAPH ---------------- ------- ---------------------- 06-AUG 13:00:00 .33 +- 2 06-AUG 14:00:00 2.24 ++--------2--- 06-AUG 15:00:00 6.67 ++++------2---------- 06-AUG 16:00:00 2.59 ++--------2---- 06-AUG 17:00:00 1.26 ++----- 2 06-AUG 18:00:00 1.38 +++---- 2 06-AUG 19:00:00 1.74 ++------- 2 06-AUG 20:00:00 .99 +---- 2 06-AUG 21:00:00 1.22 ++----- 2 06-AUG 22:00:00 1.66 ++------ 2 06-AUG 23:00:00 1.08 +---- 2 07-AUG 00:00:00 .83 +--- 2 07-AUG 01:00:00 1.74 ++------- 2 07-AUG 02:00:00 2.47 ++--------2---- 07-AUG 03:00:00 6.59 +++-------2---------- 07-AUG 04:00:00 1.95 ++++++--- 2 07-AUG 05:00:00 3.08 +++++-----2------ 1. Load Charts:1. Load Charts: ash_graph.sqlash_graph.sql
  • 68. 1. Load Charts:1. Load Charts: ash_graph.sqlash_graph.sql “-” = WAIT “+” = CPU which waits ?
  • 69. 1. Load Charts : ash_graph.sql START_TIME GRAPH -------------------- -------------------------- NOV 03 2013 22:02:00 ++---1------ NOV 03 2013 22:03:00 ++---1------ NOV 03 2013 22:04:00 +++--1--------------------- NOV 03 2013 22:05:00 +----1- NOV 03 2013 22:06:00 +++--1-- When waiting, which are the top waits ?
  • 70. 1. Load Charts:1. Load Charts: ash_graph_ash_graph_waitswaits.sql.sql Top Two Waits
  • 71. 1. Load Charts : ash_graph_waits.sql TO_CHAR( PCT1 FIRST PCT2 SECOND GRAPH -------- ---- --------------- ---- --------------- ---------------------------------------- 15 19:00 64 CPU 21 db file sequent ++o 4 15 20:00 63 CPU 19 read by other s ++++o- 4 15 21:00 31 db file sequent 24 CPU ++ooo---- 4 15 22:00 35 CPU 24 db file scatter +++++ooooooo--- 4 15 23:00 29 log file sync 25 db file sequent ++++ooooooooo-------4------------- 16 00:00 52 db file sequent 27 CPU ++++++++++oooooooooo4ooooooooooooooo-- 16 01:00 57 CPU 36 db file sequent +++++++++++oooooooo 4 16 02:00 38 db file sequent 21 CPU ++++++oooooooooooo--4--------- 16 03:00 69 db file sequent 20 CPU +++ooooooooooo 4 16 04:00 45 db file sequent 28 CPU o 4 16 05:00 58 db file sequent 24 CPU +ooo 4 16 06:00 41 db file sequent 39 CPU +oo 4 o I/O + cpu - wait OK, what about yesterday?
  • 72. 1. Load Charts History : dba_hist_active_sess_history ‱Week of ASH data !! (or more) ‱Only 1 in 10 ASH rows kept ‱Has DBID SELECT retention, snap_interval FROM wrm$_wr_control;SELECT retention, snap_interval FROM wrm$_wr_control; exec dbms_workload_repository.modify_snapshot_settings(14*24*60,60);exec dbms_workload_repository.modify_snapshot_settings(14*24*60,60); RETENTION SNAP_INTERVALRETENTION SNAP_INTERVAL ----------------- ---------------------------------- ----------------- +00014 00:00:00.0 +00000 01:00:00.0+00014 00:00:00.0 +00000 01:00:00.0 MinutesMinutesSave more history :
  • 73. 1. Load Charts: queries with history ‱ V$ACTIVE_SESSION_HISTORY - live ‱ ash_graph_waits.sql – minute buckets ‱ V$ACTIVE_SESSION_HISTORY + DBA_HIST_ACTIVE_SESS_HISTORY – combine ‱ ash_graph_waits_histash.sql – minute buckets ‱ DBA_HIST_ACTIVE_SESS_HISTORY - history only (DBID) ‱ ash_graph_histash_by_dbid.sql - input DBID ‱ ash_graph_histash_by_dbid_program.sql - input DBID and PROGRAM ‱ ash_graph_histash_by_dbid_sqlid.sql - input DBID and a SQL_ID Fast, in memory slower repositories
  • 74. 2. SQL Elapsed times (11g+ ) ASH 11g added sql_exec_id sql_exec_start SQL execs will have same above and ‱sql_id
  • 75. 2. SQL Elapsed times ‱ ash_sql_elapsed.sql longest running SQL ‱ ash_sql_elapsed_hist.sql with histogram of execution times ‱ ash_sql_elapsed_hist_longestid.sql exec id of longest query SQL_ID EXEC MX AV MIN ------------- ---------- ------- --------- ---------- 0fvrpk7476b7y 26 3068 133.1 0 1pjp66rxcj6tg 15 3106 767.7 57 SQL_ID EXECS MX MN AV 1 2 3 4 5 ------------- ---------- ---------- ---------- -------- ------ ------ ----- ----- ----- 5k7vccwjr5ahd 2653 1963 0 33.4 2623 15 8 4 3 ds8cz0fb8w147 161 2531 13 273.8 136 18 5 1 1 SQL_ID EXECS MX MN AV MAX_RUN_TIME LONGEST_SQ 1 2 3 4 5 ------------- ------ ---------- ---- -------- ----------------------------------- ---------- ------ ------ ----- ---- --- 2spgk3k0f7quz 251 29607 0 546.0 11-04-12 12:11:47 11-04-12 20:25:14 16781748 247 2 0 0 2 990m08w8xav7s 591 7681 0 52.0 11-04-13 00:39:27 11-04-13 02:47:28 16786685 587 0 0 2 2
  • 76. 2. SQL Elapsed times: ash_sql_elapsed.sql SQL_ID EXECS MX AV MIN ------------- ---------- ------- --------- ---------- 0fvrpk7476b7y 26 3068 133.1 0 1pjp66rxcj6tg 15 3106 767.7 57 8r5wuxk1dprhr 39 3510 841.0 24 0w5uu5kngyyty 21 3652 442.3 0 0hbv80w9ypy0n 161 4089 1183.9 0 71fwb4n6a92fv 49 4481 676.9 30 0bujgc94rg3fj 604 4929 24.7 0 64dqhdkkw63fd 1083 7147 7.2 0 What was the distribution?
  • 77. 2. SQL Elapsed times: ash_sql_elapsed_hist.sql SQL_ID Execs MX MN AV 1 2 3 4 5 ------------- ------ ---- --- ------ ---- ---- ---- --- --- 30hzp85f3qtxj 179 3029 29 111.0 178 0 0 0 1 Bnddu47dqmzqd 10 3138 68 732.3 7 1 1 0 1 8rua4c9agcqkb 978 820 0 72.7 830 115 21 9 2 Histogram buckets: elapsed time spread 820/5 = 164820 / 5 = 164 secs width of each bucket
  • 78. 2. SQL Elapsed times: ash_sql_elapsed_hist.sql SQL_ID Execs MX MN AV 1 2 3 4 5 ------------- ------ ---- --- ------ ---- ---- ---- --- --- 30hzp85f3qtxj 179 3029 29 111.0 178 0 0 0 1 Bnddu47dqmzqd 10 3138 68 732.3 7 1 1 0 1 8rua4c9agcqkb 978 820 0 72.7 830 115 21 9 2 820/5 = 164164 secs width of each bucket 1. 0 – 164 830executions 2. 165 – 328 115 executions 3. 329 – 492 21 executions 4. 493 – 656 9 executions 5. 657 – 820 2 executions
  • 79. 2. SQL Elapsed times: ash_sql_elapsed_hist.sql SQL_ID Execs MX MN AV 1 2 3 4 5 ------------- ------ ---- --- ------ ---- ---- ---- --- --- 30hzp85f3qtxj 179 3029 29 111.0 178 0 0 0 1 Bnddu47dqmzqd 10 3138 68 732.3 7 1 1 0 1 8rua4c9agcqkb 978 820 0 72.7 830 115 21 9 2 820/5 = 164 1. <= 164 830 executions 2. <= 328 115 executions 3. <= 492 21 executions 4. <= 656 9 executions 5. <= 820 2 executions
  • 80. 2. SQL Elapsed times: ash_sql_elapsed_hist.sql SQL_ID Execs MX MN AV 1 2 3 4 5 ------------- ------ ---- --- ------ ---- ---- ---- --- --- 30hzp85f3qtxj 179 3029 29 111.0 178 0 0 0 1 Bnddu47dqmzqd 10 3138 68 732.3 7 1 1 0 1 8rua4c9agcqkb 978 820 0 72.7 830 115 21 9 2 820/5 = 164 Outlier
  • 81. 2. SQL Elapsed Times ash_sql_elapsed_hist_longestid.sql SQL_ID Execs MX MN AV MAX_RUN_TIME LONGEST_SQ 1 2 3 4 5 ------------- ---- ------ ---- ------ ------------------------------------------- ---------- ---- ----- ---- ---- --- 30hzp85f3qtxj 179 3029 29 111.0 11-11-07 13:46:12 11-11-07 14:36:41 16777250 177 1 0 0 1 C61wk6d7ssxxc 20 1801 258 743.0 11-11-07 12:01:02 11-11-07 12:31:03 16777216 13 2 0 0 5 769uu28qm4thw 17 1427 0 479.0 11-11-07 14:04:02 11-11-07 14:27:49 16777762 10 2 2 1 2 Fuzcbdt08xjcd 74 315 33 248.0 11-11-07 12:10:05 11-11-07 12:15:20 16790567 5 4 9 5 51 Bkv51bug8ag7c 29 307 1 36.0 11-11-07 11:58:30 11-11-07 12:03:37 16790749 26 2 0 0 1 1wgsn7mmf6kqc 131 299 0 6.0 11-11-07 12:01:08 11-11-07 12:06:07 16784143 130 0 0 0 1 3b7q0hd1q8pw0 17 288 1 118.0 11-11-07 09:42:21 11-11-07 09:47:09 16794374 7 4 0 1 5 1v6yyfy630rkj 13 135 42 65.0 11-11-07 07:49:12 11-11-07 07:51:27 16816140 7 4 1 0 1 1t715k5p9uxxx 50 95 0 32.0 11-11-07 05:17:55 11-11-07 05:19:30 16777225 32 1 1 0 16 A98fbc69py0us 11 91 59 74.0 11-11-07 14:55:55 11-11-07 14:57:26 16777389 1 4 3 2 1 0g53kf4gr3vrg 25 53 32 46.0 11-11-07 15:35:31 11-11-07 15:36:24 16777588 1 0 5 14 5 OK found the longest but ‱When did it happen? ‱What was the sql_exec_id of the longest? start End
  • 82. 2. SQL Elapsed Times – lookup by SQL ID Select decode(session_state,'ON CPU','ON CPU’, event) WAIT From v$active_session_history Where sql_exec_id = 16777217 Order by sample_id WAIT ------------------------------------------------ db file parallel read db file sequential read ON CPU ON CPU ON CPU ON CPU
  • 83. 3. Wait Analysis: v$active_session_history ‱ Buffer busy waits ‱ Enqueue transaction waits
  • 84. Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Call Time -------------------------------------------- ------------ ----------- --------- buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free buffer waits 1,588 15 4.63 write complete waits 10 8 2.51 log buffer space 306 5 1.51 ------------------------------------------------------------ Buffer Busy Wait – trying to modify a block Who blocks? What object? what is the SQL? AWR fails 3. Wait Analysis: buffer busy waits
  • 85. 3. Wait Analysis: buffer busy waits OBJN OTYPE FILEN BLOCKN SQL_ID BLOCK_TYPE ------------------------- ----------- ------ ------------- ------------ 53218 BBW_INDEX_VAL_I INDEX 1 64826 97dgthz60u28d data block 1 53218 BBW_INDEX_VAL_I INDEX 1 64826 gypmcfzruu249 data block 1 53218 BBW_INDEX_VAL_I INDEX 1 64826 2vd1w5kgnfa5n data block 1 53218 BBW_INDEX_VAL_I INDEX 1 64826 3p3qncvp2juxs data block 1 53218 BBW_INDEX_VAL_I INDEX 1 64826 6avm49ys4k7t6 data block 1 ash_bbw.sql To solve Buffer Busy Waits, need ‱Object ‱ Table w/ data block => ASM or Freelists, ‱ Index w/data block => reverse or hash partition ‱File ‱ Temp files problem with extents
  • 86. 3. Wait Analysis: Joining ASH with v$waitstat select o.object_name obj, o.object_type otype, ash.SQL_ID, w.class from v$active_session_history ash, ( select rownum class#, class from v$waitstat ) w, all_objects o where event='buffer busy waits' and w.class#(+)=ash.p3 and o.object_id (+)= ash.CURRENT_OBJ# Order by sample_time; OBJ OTYPE SQL_ID CLASS ------ ------ ------------- ------------------ TOTO1 TABLE 8gz51m9hg5yuf data block TOTO1 TABLE 8gz51m9hg5yuf data block TOTO1 TABLE 8gz51m9hg5yuf segment header TOTO1 TABLE 8gz51m9hg5yuf data block
  • 87. 3. Wait Analysis: How to get Class Name select rownum n,ws.class from v$waitstat; NAME P1 P2 P3 ----------------- ----- ------ ----- buffer busy waits file# block# class# NAME P1 P2 P3 ----------------- ----- ------ ----- buffer busy waits file# block# class# select * from v$event_name where name = 'buffer busy waits' N CLASS 1 data block 2 sort block 3 save undo block 4 segment header 5 save undo header 6 free list 7 extent map 8 1st level bmb 9 2nd level bmb 10 3rd level bmb 11 bitmap block 12 bitmap index block 13 file header block 14 unused 15 system undo header 16 system undo block P3 = class#, how do we get class name?
  • 88. 3. Wait Analysis: tx enqueues Top 5 Timed Events Avg %Total ~~~~~~~~~~~~~~~~~~ wait Call Event Waits Time (s) (ms) Time ----------------------------------------- ------------ ----------- ------ ------ enq: TX - row lock contention 59 160 2714 41.8 PL/SQL lock timer 4 117 29291 30.6 CPU time 28 7.2 buffer busy waits 1,217 18 15 4.7 log file parallel write 422 11 27 3.0 Who is waiting? Who is blocking? What is the SQL? What is the row? Not in AWR report
  • 89. 3. Wait Analysis: buffer busy waits ‱ Waiter ‱ SESSION_ID ‱ SESSION_SERIAL# ‱ USER_ID ‱ Object ‱ CURRENT_OBJ# ‱ CURRENT_FILE# ‱ CURRENT_BLOCK# ‱ SQL Waiting - SQL_ID ‱ Blocker - BLOCKING_SESSION - BLOCKING_SESSION_STATUS - BLOCKING_SESSION_SERIAL# ‱ Lock Type and Mode - Event = Type (name) - P1 = Type | Mode v$active_session_historyv$active_session_history
  • 90. 3. Wait Analysis: tx enqueue waits select to_char(sample_time,’HH MI’) st, substr(event,0,20) lock_name, ash.session_id waiter, mod(ash.p1,16) lmode, ash.p2 p2, ash.p3 p3, o.object_name object, o.object_type otype, CURRENT_FILE# filen, CURRENT_BLOCK# blockn, ash.SQL_ID waiting_sql, BLOCKING_SESSION blocker from v$active_session_history ash, all_objects o where event like 'enq: T%' and o.object_id (+)= ash.CURRENT_OBJ# / Mode 6 row Mode 4 Other ash_enq.sql
  • 91. 3. Waits: enq TX 43. Waits: enq TX 4 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 Mode 4, foreign key Mode 4, unique key Mode 4, bitmap No Object File or Block
  • 92. 3. Waits: enq TX 43. Waits: enq TX 4 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 Mode 4, foreign key Mode 4, unique key Mode 4, bitmap Child Table Blocker SID=1
  • 93. 3. Waits: enq TX 43. Waits: enq TX 4 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 10:39 enq: TX - row lock c 141 4 655406 6672 -1 0 0 bjvx94vnxtxgv 158 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 10:41 enq: TX - row lock c 144 4 179681 7074 CHILD TABLE 1 60954 ahm7c9rupbz9r 1 ST EVENT SID LM P2 P3 OBJ OTYPE FN BLOCKN SQL_ID BSID ----- ---------------------- --- --- ------ ---- ----- ----- --- ------ -- 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 10:41 enq: TX - row lock c 143 4 966081 4598 I1 INDEX 0 0 azav296xxqcjx 144 Mode 4, foreign key Mode 4, unique key Mode 4, bitmap Obect Index No File or Block
  • 94. 4. Top A.Procedure ‱ ash_top_procedure.sql A.Session ‱ ash_top_session.sql - wait, I/O and CPU time A.SQL i.ash_top_sql.sql - wait, I/O and CPU time ii.ash_top_sql_w_top_obj.sql - with top OBJ per SQL
  • 95. 10.2.0.3 ASH added10.2.0.3 ASH added PLSQL_ENTRY_OBJECT_ID PLSQL_ENTRY_SUBPROGRAM_ID PLSQL_OBJECT_ID PLSQL_SUBPROGRAM_ID 9509/19/16 ALL_PROCEDURES viewALL_PROCEDURES view object_id = plsql_object_idobject_id = plsql_object_id subprogram_id = plsql_subprogram_idsubprogram_id = plsql_subprogram_id 4A. Top: PROCEDURE ash_top_procedure.sql
  • 96. 4A. Top: PROCEDURE ash_top_procedure.sql COUNT(*) SQL_ID calling_code --------- ------------- ------------------------------------------ 2 1xxksrhwtz3zf ORDERENTRY.NEWORDER => DBMS_RANDOM.VALUE 2 1xxksrhwtz3zf ORDERENTRY.NEWORDER => DBMS_LOCK.SLEEP 13 1xxksrhwtz3zf ORDERENTRY.NEWORDER 16 0bzhqhhj9mpaa ORDERENTRY.NEWCUSTOMER 45 41zu158rqf4kf ORDERENTRY.BROWSEANDUPDATEORDERS
  • 97. 4B. Top: SESSION ash_top_session.sql STATUS SID NAME PROGRAM CPU WAITING IO TOTAL ------------ ----- ------------ ------------------------- ----- ------- ----- ------ DISCONNECTED 54 SYS oracle@source (J000) 1227 540 152 1919 DISCONNECTED 57 SYS oracle@source (J001) 725 160 18 903 DISCONNECTED 71 SYS sqlplus@source (TNS V1-V3 535 60 36 631 DISCONNECTED 67 SYSTEM LAB128.exe 187 182 148 517 CONNECTED 10 SYS oracle@source (DBW0) 267 171 0 438 CONNECTED 11 SYS oracle@source (LGWR) 10 357 0 367 DISCONNECTED 44 SYS sqlplus@source (TNS V1-V3 103 158 15 276 CONNECTED 53 SYSTEM JDBC Thin Client 129 33 0 162 CONNECTED 36 SYSMAN OMS 6 114 0 120
  • 98. 4Ci. Top: SQL ash_top_sql.sql SQL_ID PLAN_HASH TYPE CPU WAIT IO TOTAL ------------- ---------- ---------- ------- ----- ------- ------ fdtr6ds1nxm5r 1269593971 SELECT 9345 9 84277 93631 5aa7r665a07n6 3761618565 SELECT 82040 6 850 82896 2x8kgb0s9q1zh 0 PL/SQL EXE 75967 1575 1186 78728 4qraawku9303p 2256852101 SELECT 46995 45 3196 50236 3h7agx5ndadrf 1006614515 SELECT 2068 1 24914 26983 8p2cyq3gdgau5 2108428761 SELECT 22345 489 313 23147 7vpqr5zhjm08v 4089415459 SELECT 422 0 19655 20077 7sf51jrzq6y3c 485652470 INSERT 1727 1418 15471 18616 6cy7mc2kj0u4z 3041431021 SELECT 14272 0 0 14272
  • 99. 4Cii. Top: SQL ash_top_sql_w_top_obj.sql SQL_ID SQLPLANHASH AUD_ACT CPU WAIT IO TOTAL PCT_IO TOP_OBJ ------------- ----------- ------- ---- ----- ---- ------- ------- -------------- dc9fkz2t3b9p8 1909389838 SELECT 3685 0 5 3690 60 PS_F_ABS_EXPTN_CA 30hzp85f3qtxj 3802366046 SELECT 568 0 921 1489 100 PS_TAX_BALANCE c61wk6d7ssxxc 702141750 SELECT 117 6 1364 1487 98 PS_PAY_DEDUCTION c3y6kdm1uzkb7 2395607145 SELECT 39 0 1375 1414 87 PS_PAY_CHECK 6wsy8rpd0bw26 252818247 SELECT 1363 0 14 1377 64 PS_F_TIALAST_HIST
  • 100. 5. I/O – lots you can do I/O by ‱SQL ‱Object ‱File & Table Space ‱Parallel Query master and slaves ‱By type of I/O ‱ Db file sequential read ‱ Db file scattered read ‱ Direct path read ‱ Direct path read temp ‱ Direct path write ‱ Direct path write temp
  • 101. 5. I/O : iosql.sql I/O by SQL and top Objects for that sql AAS SQL_ID % OBJ TABLESPACE ----- ------------- --- --------------- ---------- .18 0yas01u2p9ch4 6 ITEM_PRODUCT_IX SOEINDEX 6 ORDER_ITEMS_UK SOEINDEX 88 ITEM_ORDER_IX SOEINDEX .32 6v6gm0fd1rgrz 6 WAIT_OBJECTS SYSTEM 94 UNDO UNDOTBS1
  • 102. Extra : Latency 1. latency_eventmetric.sql - Now ‱ 60 seconds avg ‱ V$EVENTMETRIC 1. latency_system_event.sql - Historic ‱ Hour average ‱ DBA_HIST_SYSTEM_EVENT Can’t get latency from: avg(TIME_WAITED) !
  • 103. latency: latency_eventmetric.sql NAME WAIT_COUNT AVGMS ------------------------- ---------- ---------- log file parallel write 12 3.213 log file sync 6 4.682 db file sequential read 0 0 db file scattered read 0 0 direct path read 0 0 direct path read temp 0 0 direct path write 0 0 direct path write temp 0 0 Last 60 seconds
  • 104. latency: last 60 seconds select -- m.intsize_csec, n.name , round(m.time_waited,3) time_waited, m.wait_count, round(10*m.time_waited/nullif(m.wait_count,0),3) avgms from v$eventmetric m, v$event_name n where m.event_id=n.event_id and n.name in ( 'db file sequential read', 'db file scattered read', 'direct path read', 'direct path read temp', 'direct path write', 'direct path write temp', 'log file sync', 'log file parallel write' I/O Events v$event_metric only had event ids  Join to v$event_name latency_eventmetric.sql
  • 105. Latency: each hour latency_system_event.sql BTIME AVG_MS ------------------------ ---------- 01-MAY-13 23:00 .199 02-MAY-13 00:00 .023 02-MAY-13 01:00 .031 02-MAY-13 02:00 .006 02-MAY-13 03:00 .017 02-MAY-13 04:00 .015 02-MAY-13 05:00 .013 02-MAY-13 06:00 .019 02-MAY-13 07:00 .017 DBA_HIST_SYSTEM_EVENT DBA_HIST_SNAPSHOT Input event name
  • 106. Oramon.sh Connected, starting collect at Wed Dec 5 14:59:24 EST 2012 starting stats collecting single block logfile write multi block direct read direct read temp direct write temp ms IOP/s ms IOP/s ms IOP/s ms IOP/s ms IOP/s ms IOP/s 3.53 .72 16.06 .17 4.64 .00 115.37 3.73 .00 0 1.66 487.33 2.66 138.50 4.84 33.00 .00 .00 0 1.71 670.20 3.14 195.00 5.96 42.00 .00 .00 0 2.19 502.27 4.61 136.82 10.74 27.00 .00 .00 0 1.38 571.17 2.54 177.67 4.50 20.00 .00 .00 0 single block logfile write multi block direct read direct read temp direct write temp ms IOP/s ms IOP/s ms IOP/s ms IOP/s ms IOP/s ms IOP/s 3.22 526.36 4.79 135.55 .00 .00 .00 0 2.37 657.20 3.27 192.00 .00 .00 .00 0 1.32 591.17 2.46 187.83 .00 .00 .00 0 2.23 668.60 3.09 204.20 .00 .00 .00 .00 0 https://github.com/khailey/oramon Usage: oramon.sh [username] [password] [host] [sid] <port=1521> <runtime=3600>
  • 107. Summary Load on database = AAS = DB Time/elapsed ~ = count(*)/elapsed Select count(*)/{time period in seconds} AAS from ASH where {criteria} and {time period}; Custom queries can give detailed info on ‱ ASH Graph ‱ SQL excution, max, min ‱ Top SQL, Session, Procedure, Object ‱ IO by SQL, Object, Tablespace ‱ Detailed wait analysis – BBW, Enqueues etc https://github.com/khailey/ashmasters
  • 108. END
  • 109. 1. Load Charts: basics, fake query1. Load Charts: basics, fake query Select pseudo columns start_time , bucket_id , count(cpu) , count(waits) from v$active_session_history ash START_TIME BUCKET_ID CPU WAITS ---------- ------------- ---------- ---------- 25-NOV-13 55 397 28231 25-NOV-13 56 80 5631
  • 110. 1. Load Charts: basics, fake query1. Load Charts: basics, fake query Select pseudo columns start_time , bucket_id , count(cpu) , count(waits) from v$active_session_history ash START_TIME BUCKET_ID CPU WAITS ---------- ------------- ---------- ---------- 25-NOV-13 55 397 28231 25-NOV-13 56 80 5631
  • 111. 1. Load Charts: components1. Load Charts: components BUCKET ID ID = truncate ( seconds in date / seconds in bucket ) truncate (0/5) = 0 truncate (1/5) = 0 truncate (2/5) = 0 truncate (3/5) = 0 truncate (4/5) = 0 truncate (5/5) = 1 truncate (6/5) = 1 truncate (7/5) = 1 truncate (8/5) = 1 truncate (9/5) = 1 0- 5 secs = 0 6-10 secs = 1
  • 112. 1. Load Charts: components1. Load Charts: components BUCKET ID ID = truncate ( seconds in date / seconds in bucket ) (( Julian days * seconds in a day ) + (seconds today)) / bucket size in seconds
  • 113. 1. Load Charts: components1. Load Charts: components BUCKET ID ID = truncate ( seconds in date / seconds in bucket ) (( Julian days * seconds in a day ) + (seconds today)) / bucket size in seconds trunc((to_char(sample_time,'J')*(24*60*60)+ to_char(sample_time,'SSSSS'))/&v_secs)
  • 114. 1. Load Charts: components1. Load Charts: components BUCKET ID ID = truncate ( seconds in date / seconds in bucket ) (( Julian days * seconds in a day ) + (seconds today)) / bucket size in seconds trunc((to_char(sample_time,'J')*(24*60*60)+ to_char(sample_time,'SSSSS'))/&v_secs) Counts , sum(decode(session_state,'ON CPU',1,0)) cpu , sum(decode(session_state,'WAITING',1,0)) waits
  • 115. 1. Load Charts: real query1. Load Charts: real query select trunc((to_char(sample_time,'J')*(24*60*60)+ to_char(sample_time,'SSSSS'))/&v_secs) id , sum(decode(session_state,'ON CPU',1,0)) cpu , sum(decode(session_state,'WAITING',1,0)) waits from v$active_session_history ash -- dba_hist_active_sess_history group by trunc((to_char(sample_time,'J')*(24*60*60)+ to_char(sample_time,'SSSSS'))/&v_secs) ID CPU WAITS ---------- ---------- ---------- 4.2432E+10 1 4 4.2432E+10 9 18 4.2432E+10 8 25 4.2432E+10 9 26 Group by ID ID Waits CPU
  • 116. 1. Load Charts: how do you draw a load1. Load Charts: how do you draw a load line?line? rpad('+',round(( cpu/&v_secs)*&v_bars),'+’) || rpad('-',round((waits/&v_secs)*&v_bars),'-') || If the value of 1 AAS is 5 characters wide then +++++----- Above is AAS of 2 (1 wait, 1 CPU) AAS Width of AAS Def v_secs=60 -- size bucket in seconds Def v_bars=5 -- size of one AAS
  • 117. 1. Load Charts: basics : graphics1. Load Charts: basics : graphics rpad('+',round(( cpu/&v_secs)*&v_bars),'+’) || rpad('-',round((waits/&v_secs)*&v_bars),'-') || rpad(' ',p.value * &v_bars,' '), -- blanks +++++----- 1 +++++----- +++++1----- Trick, add in the number cores 1.Draw full line, cut at core # 2.Add core # 3.Draw full line again cut before first core # || || What about a yard stick ?
  • 118. 1. Load Charts: graphics1. Load Charts: graphics Def v_bars=5 -- size of one AAS in characters Def v_graph=80 substr( substr( rpad('+',round((cpu*&v_bars)/&v_secs),'+') || rpad('-',round((waits*&v_bars)/&v_secs),'-') || rpad(' ',p.value * &v_bars,' '), 0,(p.value * &v_bars)) || p.value || substr( rpad('+',round((cpu*&v_bars)/&v_secs),'+') || rpad('-',round((waits*&v_bars)/&v_secs),'-') || rpad(' ',p.value * &v_bars,' '), (p.value * &v_bars)) ,0,&v_graph) 2nd half of line Add core count First half of line +++++----- 1 +++++-----
  • 119. 1. Load Charts: basics1. Load Charts: basicsselect to_char(to_date( trunc((id*&v_secs)/ (24*60*60)) || ' ' || -- Julian days mod((id*&v_secs), (24*60*60)) -- seconds in the day , 'J SSSSS' ), 'MON DD YYYY HH24:MI:SS') start_time, substr( substr( rpad('+',round(( cpu/&v_secs)*&v_bars),'+’) || rpad('-',round((waits/&v_secs)*&v_bars),'-') || rpad(' ',p.value * &v_bars,' ’) ,0,(p.value * &v_bars)) || p.value || substr( rpad('+',round(( cpu/&v_secs)*&v_bars),'+’) || rpad('-',round((waits/&v_secs)*&v_bars),'-') || rpad(' ',p.value * &v_bars,' '), (p.value * &v_bars)) ,0,&v_graph) graph from ( select trunc((to_char(sample_time,'J')*(24*60*60)+to_char(sample_time,'SSSSS'))/&v_secs) id , sum(decode(session_state,'ON CPU',1,0)) cpu , sum(decode(session_state,'WAITING',1,0)) waits from v$active_session_history ash group by trunc((to_char(sample_time,'J')*(24*60*60)+to_char(sample_time,'SSSSS'))/&v_secs) ) aveact, v$parameter p where p.name='cpu_count' order by id / Bar Date (from ID) ID WAITS CPU RAW Graphics
  • 120. 1. Load Charts : events ash_graph_waits.sql BUCKET_ID EVENT EVENT_COUNT ---------- ------------------------------- ----------- 3537592817 control file heartbeat 2 3537592818 ADR block file read 1 3537592818 ARCH wait for process start 3 3 3537592818 ON CPU 9 3537592818 control file parallel write 4 Select trunc((to_char(sample_time,'J')*(24*60*60)+to_char(sample_time,'SSSSS'))/&v_secs) Bucket_id , decode(session_state,'ON CPU','ON CPU',ash.event) event , count(*) event_count From v$active_session_history ash group by trunc((to_char(sample_time,'J')*(24*60*60)+to_char(sample_time,'SSSSS'))/&v_secs) , decode(session_state,'ON CPU','ON CPU',ash.event)
  • 121. 1. Load Charts : per bucket, rank the waits ID EVENT RANK PCT CPU WAITS ---------- ----------------------------- ---- --- ---- ----- 3537592818 ON CPU 1 .290 9 0 3537592818 db file sequential read 2 .161 0 5 3537592818 log file switch (checkpoint i 3 .161 0 5 3537592818 control file parallel write 4 .129 0 4 3537592818 ARCH wait for process start 5 .096 0 3 3537592818 ADR block file read 6 .032 0 1 select id , event , row_number() over ( partition by id order by event_count desc ) rank , ratio_to_report( event_count ) over ( partition by id ) pct , sum(decode(event,'ON CPU',event_total,0)) cpu , sum(decode(event,'ON CPU',0,event_total)) waits from ( Previous Query ) chunks group by id, event, event_total Rank Rank %
  • 122. 1. Load Charts : raw data ash_graph_waits.sql select id , round(max(decode(top.rank,1,pct,null)),2) fpct , max( decode(top.rank,1,decode(top.event,'ON CPU','CPU',event),null)) first , round(max(decode(top.rank,2,pct,null)),2) spct , max( decode(top.rank,2,decode(top.event,'ON CPU','CPU',event),null)) second , sum(waits) waits , sum(cpu) cpu 
 ID FPCT FIRST SPCT SECOND WAITS CPU ---------- ----- --------------- ----- --------------- ---------- ------ 3537592817 1.00 control file he 2 0 3537592818 .29 CPU .16 db file sequent 22 9 3537592819 .72 control file pa .14 CPU 37 6 3537592820 .75 CPU .25 control file pa 1 3 3537592821 .91 CPU .09 direct path rea 1 10 3537592823 .75 CPU .25 db file paralle 1 3
  • 123. 2. SQL Elapsed times : raw data select sql_id, sql_exec_id, -- SAMPLE_TIME - START TIME (cast(sample_time as date) – cast(sql_exec_start as date)) * (60*60*24) secs from v$active_session_history where sql_exec_id is not null seconds query had been running = Sample_time – sql_exec_start now select max seconds for each sql_exec_id Seconds running id SQL_ID SQL_EXEC_ID SECS ------------- ----------- ------ acc988uzvjmmt 16777220 3 acc988uzvjmmt 16777220 2 acc988uzvjmmt 16777220 1 acc988uzvjmmt 16777220 0
  • 124. 2. SQL Elapsed times select sql_id, sql_exec_id, max(tm) tm from ( previous query ) Final execution time is max of all execs SQL_ID SQL_EXEC_ID TM ------------- ----------- ---------- acc988uzvjmmt 16777220 3 acc988uzvjmmt 16777335 15 acc988uzvjmmt 16779830 1 acc988uzvjmmt 16781502 7
  • 125. 2. SQL Elapsed times select sql_id, count(*), max(tm) mx, avg(tm) av, min(tm) min from ( previous query ) each SQL_ID ‱max ‱min ‱average execution times SQL_ID COUNT(*) MX AV MIN ------------- ---------- ------- --------- ---------- acc988uzvjmmt 4 15 6.5 1
  • 126. 2. SQL Elapsed times: ash_sql_elapsed.sql SQL_ID Executions MX AV MIN ------------- ---------- ------- --------- ---------- 0fvrpk7476b7y 26 3068 133.1 0 1pjp66rxcj6tg 15 3106 767.7 57 8r5wuxk1dprhr 39 3510 841.0 24 0w5uu5kngyyty 21 3652 442.3 0 0hbv80w9ypy0n 161 4089 1183.9 0 71fwb4n6a92fv 49 4481 676.9 30 0bujgc94rg3fj 604 4929 24.7 0 64dqhdkkw63fd 1083 7147 7.2 0 990m08w8xav7s 591 7681 51.8 0 2n5369dsuvn5a 16 10472 5726.8 303 2spgk3k0f7quz 251 29607 546.1 0 36pd759xym9tc 12 37934 23861.9 1391 497wh6n7hu14f 49 69438 5498.2 0
  • 127. iosql.sql select sum(cnt) over ( partition by io.sql_id order by sql_id ) tcnt, io.sql_id, io.cnt cnt, io.aas aas, io.objn objn, io.obj obj, io.p1 p1, f.tablespace_name tablespace_name from ( select sql_id, count(*) cnt, round(count(*)/(&v_minutes*60),2) aas, CURRENT_OBJ# objn, nvl(o.object_name,decode(CURRENT_OBJ#,-1,0,CURRENT_OBJ#)) obj, o.object_type otype, ash.p1 from v$active_session_history ash ,all_objects o where ( event like 'db file s%' or event like 'direct%' ) and o.object_id (+)= ash.CURRENT_OBJ# and sample_time > sysdate - &v_minutes/(60*24) group by CURRENT_OBJ#,o.object_name ,o.object_type , ash.p1, sql_id ) io, dba_data_files f where f.file_id = io.p1 Order by tcnt, io.sql_id, io.cnt /

Editor's Notes

  • #2: Have you ever been hiking in the Yosemite? I went hiking in Yosemite by myself. I was friends with one of the park rangers in Yosemite. She love to hike and she always hiked off trail. She told me about a beautiful lake off trail that would be wonderful to hike to. The lake was just on the other side of the mountain crest from where we were. She gave me a topographic map to use. Have you ever used a topographic map? It’s a map that shows elevations with contour lines. Each line denotes say 50’ of elevation change. The more lines together closer together the steeper the terrain. I used this map to plot out a course to the lake that didn’t have cliffs or other obstacles. I managed to make it to the top of the mountain crest and when there I could see the beautiful lake below way off any trail. From where I was it was a steep descent of 300 feet to the lake. All approaches looked the same on the topographic map. As I approached the lake it got steeper and steeper and steeper. I finally found my self on a vertial face with no way to go down father And unsure if I could even climb back up. I was stuck. Then I thought “if I got here with a big back pack on, I can certainly climb back up without it!” I gingerly undid the clips and slid my backpack off my back. Swoosh It went down the rock face. Unbeknownst to me, there was a snow ramp at the bottom of the clift left over from the winter. My pack hit the snow ramp at full force and went shooting out into the middle of the lake. Here I was like at 8pm at on a late summer’s dusk in the mountains at 11’000 feet way off train after a day of hiking and my backpack with my foot, shelter, warm clothing was in the middle of a lake 100-200 feet below. Lucky for me my backpack floated and before long had floated to the side of the lake. I went back up tried a different path down and made it. But it just goes to show that the user interface of this particular topographic map didn’t distinguish between an impenetrable 200 feet and a navigat-able 200 feet drop. Hi, my names Kyle Hailey, and I love working with user interface s that help me navigate the world.
  • #8: This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another.
  • #9: This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another.
  • #31: This presentation presented a challenge because many of the concepts were like the chicken and the egg. In order to explain one I needed the other. I’ve tried to keep the subjects as linear as possible but sometimes I interweave from one to another.
  • #32: WAIT_TIME is time of last wait, &amp;gt; 0 , means on CPU TIME_WAITED is the actual wait time
  • #43: Select session_id, session_serial# , count(*) , round((count(*)*100)/(5*60),0) &amp;quot;%&amp;quot;, lpad(&amp;apos;*&amp;apos;,round((count(*)*10)/(5*60),0),&amp;apos;*&amp;apos;) &amp;quot;Bar&amp;quot; from v$active_session_history where session_state= &amp;apos;ON CPU&amp;apos; and SAMPLE_TIME &amp;gt; sysdate - (5/(24*60)) group by session_id, session_serial# order by count(*) desc /
  • #45: Select session_id, session_serial# , count(*) , round((count(*)*100)/(5*60),0) &amp;quot;%&amp;quot;, lpad(&amp;apos;*&amp;apos;,round((count(*)*10)/(5*60),0),&amp;apos;*&amp;apos;) &amp;quot;Bar&amp;quot; from v$active_session_history where session_state= &amp;apos;ON CPU&amp;apos; and SAMPLE_TIME &amp;gt; sysdate - (5/(24*60)) group by session_id, session_serial# order by count(*) desc /
  • #46: w
  • #47: Graphics – humans can process 400,000 times as much data graphically as textually Think of a map of the US and all the counties with birth rates colored from low to high. It’s a massive about of information textually but it can all be seen on one page of a map Select session_id, session_serial# , count(*) , round((count(*)*100)/(5*60),0) &amp;quot;%&amp;quot;, lpad(&amp;apos;*&amp;apos;,round((count(*)*10)/(5*60),0),&amp;apos;*&amp;apos;) &amp;quot;Bar&amp;quot; from v$active_session_history where session_state= &amp;apos;ON CPU&amp;apos; and SAMPLE_TIME &amp;gt; sysdate - (5/(24*60)) group by session_id, session_serial# order by count(*) desc /
  • #87: col objn for a25 select --ash.p1, --ash.p2, --ash.p3, CURRENT_OBJ#||&amp;apos; &amp;apos;||o.object_name objn, o.object_type otype, CURRENT_FILE# filen, CURRENT_BLOCK# blockn, ash.SQL_ID, w.class from v$active_session_history ash, ( select rownum class#, class from v$waitstat ) w, all_objects o where event=&amp;apos;buffer busy waits&amp;apos; and w.class#(+)=ash.p3 and o.object_id (+)= ash.CURRENT_OBJ# Order by sample_time /
  • #91: col event for a22 col block_type for a18 col objn for a18 col otype for a10 col fn for 99 col sid for 9999 col bsid for 9999 col lm for 99 col p3 for 99999 col blockn for 99999 select to_char(sample_time,&amp;apos;HH:MI&amp;apos;) st, substr(event,0,20) event, ash.session_id sid, mod(ash.p1,16) lm, ash.p2, ash.p3, nvl(o.object_name,ash.current_obj#) objn, substr(o.object_type,0,10) otype, CURRENT_FILE# fn, CURRENT_BLOCK# blockn, ash.SQL_ID, BLOCKING_SESSION bsid --,ash.xid from v$active_session_history ash, all_objects o where event like &amp;apos;enq: TX %&amp;apos; and o.object_id (+)= ash.CURRENT_OBJ# and sample_time &amp;gt; sysdate - &amp;1/(60*24) Order by sample_time /
  • #112: select to_char(start_time,&amp;apos;DD HH24:MI:SS&amp;apos;), samples, substr(substr(rpad(&amp;apos;+&amp;apos;,round((cpu*&amp;v_bars)/&amp;v_secs),&amp;apos;+&amp;apos;) || rpad(&amp;apos;-&amp;apos;,round((waits*&amp;v_bars)/&amp;v_secs),&amp;apos;-&amp;apos;) || rpad(&amp;apos; &amp;apos;,p.value * &amp;v_bars,&amp;apos; &amp;apos;),0,(p.value * &amp;v_bars)) || p.value || substr(rpad(&amp;apos;+&amp;apos;,round((cpu*&amp;v_bars)/&amp;v_secs),&amp;apos;+&amp;apos;) || rpad(&amp;apos;-&amp;apos;,round((waits*&amp;v_bars)/&amp;v_secs),&amp;apos;-&amp;apos;) || rpad(&amp;apos; &amp;apos;,p.value * &amp;v_bars,&amp;apos; &amp;apos;),(p.value * &amp;v_bars),10) ,0,30) graph from ( select start_time , max(samples) samples , sum(decode(event,&amp;apos;ON CPU&amp;apos;,total,0)) cpu , sum(decode(event,&amp;apos;ON CPU&amp;apos;,0,total)) waits from ( select to_date(to_char(sample_time,&amp;apos;YYMMDD&amp;apos;)||&amp;apos; &amp;apos;||trunc(to_char(sample_time,&amp;apos;SSSSS&amp;apos;)/&amp;v_secs)*&amp;v_secs,&amp;apos;YYMMDD SSSSS&amp;apos;) start_time , to_char(sample_time,&amp;apos;YYMMDD&amp;apos;)||trunc(to_char(sample_time,&amp;apos;SSSSS&amp;apos;)/&amp;v_secs) id , decode(ash.session_state,&amp;apos;ON CPU&amp;apos;,&amp;apos;ON CPU&amp;apos;,ash.event) event , sum(decode(session_state,&amp;apos;ON CPU&amp;apos;,1,decode(session_type,&amp;apos;BACKGROUND&amp;apos;,0,1))) total , (max(sample_id)-min(sample_id)+1) samples from v$active_session_history ash group by trunc(to_char(sample_time,&amp;apos;SSSSS&amp;apos;)/&amp;v_secs) , to_char(sample_time,&amp;apos;YYMMDD&amp;apos;) , decode(ash.session_state,&amp;apos;ON CPU&amp;apos;,&amp;apos;ON CPU&amp;apos;,ash.event) ) chunks group by id, start_time ) aveact, v$parameter p where p.name=&amp;apos;cpu_count&amp;apos; order by start_time /
  • #113: select to_char(start_time,&amp;apos;DD HH24:MI:SS&amp;apos;), samples, substr(substr(rpad(&amp;apos;+&amp;apos;,round((cpu*&amp;v_bars)/&amp;v_secs),&amp;apos;+&amp;apos;) || rpad(&amp;apos;-&amp;apos;,round((waits*&amp;v_bars)/&amp;v_secs),&amp;apos;-&amp;apos;) || rpad(&amp;apos; &amp;apos;,p.value * &amp;v_bars,&amp;apos; &amp;apos;),0,(p.value * &amp;v_bars)) || p.value || substr(rpad(&amp;apos;+&amp;apos;,round((cpu*&amp;v_bars)/&amp;v_secs),&amp;apos;+&amp;apos;) || rpad(&amp;apos;-&amp;apos;,round((waits*&amp;v_bars)/&amp;v_secs),&amp;apos;-&amp;apos;) || rpad(&amp;apos; &amp;apos;,p.value * &amp;v_bars,&amp;apos; &amp;apos;),(p.value * &amp;v_bars),10) ,0,30) graph from ( select start_time , max(samples) samples , sum(decode(event,&amp;apos;ON CPU&amp;apos;,total,0)) cpu , sum(decode(event,&amp;apos;ON CPU&amp;apos;,0,total)) waits from ( select to_date(to_char(sample_time,&amp;apos;YYMMDD&amp;apos;)||&amp;apos; &amp;apos;||trunc(to_char(sample_time,&amp;apos;SSSSS&amp;apos;)/&amp;v_secs)*&amp;v_secs,&amp;apos;YYMMDD SSSSS&amp;apos;) start_time , to_char(sample_time,&amp;apos;YYMMDD&amp;apos;)||trunc(to_char(sample_time,&amp;apos;SSSSS&amp;apos;)/&amp;v_secs) id , decode(ash.session_state,&amp;apos;ON CPU&amp;apos;,&amp;apos;ON CPU&amp;apos;,ash.event) event , sum(decode(session_state,&amp;apos;ON CPU&amp;apos;,1,decode(session_type,&amp;apos;BACKGROUND&amp;apos;,0,1))) total , (max(sample_id)-min(sample_id)+1) samples from v$active_session_history ash group by trunc(to_char(sample_time,&amp;apos;SSSSS&amp;apos;)/&amp;v_secs) , to_char(sample_time,&amp;apos;YYMMDD&amp;apos;) , decode(ash.session_state,&amp;apos;ON CPU&amp;apos;,&amp;apos;ON CPU&amp;apos;,ash.event) ) chunks group by id, start_time ) aveact, v$parameter p where p.name=&amp;apos;cpu_count&amp;apos; order by start_time /
  • #114: select to_char(start_time,&amp;apos;DD HH24:MI:SS&amp;apos;), samples, substr(substr(rpad(&amp;apos;+&amp;apos;,round((cpu*&amp;v_bars)/&amp;v_secs),&amp;apos;+&amp;apos;) || rpad(&amp;apos;-&amp;apos;,round((waits*&amp;v_bars)/&amp;v_secs),&amp;apos;-&amp;apos;) || rpad(&amp;apos; &amp;apos;,p.value * &amp;v_bars,&amp;apos; &amp;apos;),0,(p.value * &amp;v_bars)) || p.value || substr(rpad(&amp;apos;+&amp;apos;,round((cpu*&amp;v_bars)/&amp;v_secs),&amp;apos;+&amp;apos;) || rpad(&amp;apos;-&amp;apos;,round((waits*&amp;v_bars)/&amp;v_secs),&amp;apos;-&amp;apos;) || rpad(&amp;apos; &amp;apos;,p.value * &amp;v_bars,&amp;apos; &amp;apos;),(p.value * &amp;v_bars),10) ,0,30) graph from ( select start_time , max(samples) samples , sum(decode(event,&amp;apos;ON CPU&amp;apos;,total,0)) cpu , sum(decode(event,&amp;apos;ON CPU&amp;apos;,0,total)) waits from ( select to_date(to_char(sample_time,&amp;apos;YYMMDD&amp;apos;)||&amp;apos; &amp;apos;||trunc(to_char(sample_time,&amp;apos;SSSSS&amp;apos;)/&amp;v_secs)*&amp;v_secs,&amp;apos;YYMMDD SSSSS&amp;apos;) start_time , to_char(sample_time,&amp;apos;YYMMDD&amp;apos;)||trunc(to_char(sample_time,&amp;apos;SSSSS&amp;apos;)/&amp;v_secs) id , decode(ash.session_state,&amp;apos;ON CPU&amp;apos;,&amp;apos;ON CPU&amp;apos;,ash.event) event , sum(decode(session_state,&amp;apos;ON CPU&amp;apos;,1,decode(session_type,&amp;apos;BACKGROUND&amp;apos;,0,1))) total , (max(sample_id)-min(sample_id)+1) samples from v$active_session_history ash group by trunc(to_char(sample_time,&amp;apos;SSSSS&amp;apos;)/&amp;v_secs) , to_char(sample_time,&amp;apos;YYMMDD&amp;apos;) , decode(ash.session_state,&amp;apos;ON CPU&amp;apos;,&amp;apos;ON CPU&amp;apos;,ash.event) ) chunks group by id, start_time ) aveact, v$parameter p where p.name=&amp;apos;cpu_count&amp;apos; order by start_time /
  • #115: select to_char(start_time,&amp;apos;DD HH24:MI:SS&amp;apos;), samples, substr(substr(rpad(&amp;apos;+&amp;apos;,round((cpu*&amp;v_bars)/&amp;v_secs),&amp;apos;+&amp;apos;) || rpad(&amp;apos;-&amp;apos;,round((waits*&amp;v_bars)/&amp;v_secs),&amp;apos;-&amp;apos;) || rpad(&amp;apos; &amp;apos;,p.value * &amp;v_bars,&amp;apos; &amp;apos;),0,(p.value * &amp;v_bars)) || p.value || substr(rpad(&amp;apos;+&amp;apos;,round((cpu*&amp;v_bars)/&amp;v_secs),&amp;apos;+&amp;apos;) || rpad(&amp;apos;-&amp;apos;,round((waits*&amp;v_bars)/&amp;v_secs),&amp;apos;-&amp;apos;) || rpad(&amp;apos; &amp;apos;,p.value * &amp;v_bars,&amp;apos; &amp;apos;),(p.value * &amp;v_bars),10) ,0,30) graph from ( select start_time , max(samples) samples , sum(decode(event,&amp;apos;ON CPU&amp;apos;,total,0)) cpu , sum(decode(event,&amp;apos;ON CPU&amp;apos;,0,total)) waits from ( select to_date(to_char(sample_time,&amp;apos;YYMMDD&amp;apos;)||&amp;apos; &amp;apos;||trunc(to_char(sample_time,&amp;apos;SSSSS&amp;apos;)/&amp;v_secs)*&amp;v_secs,&amp;apos;YYMMDD SSSSS&amp;apos;) start_time , to_char(sample_time,&amp;apos;YYMMDD&amp;apos;)||trunc(to_char(sample_time,&amp;apos;SSSSS&amp;apos;)/&amp;v_secs) id , decode(ash.session_state,&amp;apos;ON CPU&amp;apos;,&amp;apos;ON CPU&amp;apos;,ash.event) event , sum(decode(session_state,&amp;apos;ON CPU&amp;apos;,1,decode(session_type,&amp;apos;BACKGROUND&amp;apos;,0,1))) total , (max(sample_id)-min(sample_id)+1) samples from v$active_session_history ash group by trunc(to_char(sample_time,&amp;apos;SSSSS&amp;apos;)/&amp;v_secs) , to_char(sample_time,&amp;apos;YYMMDD&amp;apos;) , decode(ash.session_state,&amp;apos;ON CPU&amp;apos;,&amp;apos;ON CPU&amp;apos;,ash.event) ) chunks group by id, start_time ) aveact, v$parameter p where p.name=&amp;apos;cpu_count&amp;apos; order by start_time /
  • #124: timestamp