UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian
11 likes6,743 views
The document discusses the architecture and functioning of Oracle Clusterware 2.0, detailing its role within Oracle RAC and the components involved such as node membership, evictions, and resource management. It includes topics on clusterware architecture, startup sequences, and troubleshooting strategies. Additionally, the document covers specific features and functionalities like the Oracle Cluster Registry and behaviors during interconnect and voting disk failures.
![Startup in Linux & Unix
[root@cheese2 ~]# ps -fe | grep 'init.' | grep -v grep
root 4283 1 0 02:52 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
[root@cheese2 ~]# tail -1 /etc/inittab
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
[root@cheese2 ~]# ls -l /etc/rc3.d/*ohasd*
lrwxrwxrwx 1 root root 17 Sep 15 02:02 /etc/rc3.d/K15ohasd -> /etc/init.d/ohasd
lrwxrwxrwx 1 root root 17 Sep 15 02:02 /etc/rc3.d/S96ohasd -> /etc/init.d/ohasd
22 © 2009/2010 Pythian](https://image.slidesharecdn.com/pythian-aug12-uthoc2-130212141227-phpapp01/85/UTHOC2-Under-The-Hood-of-Oracle-Clusterware-2-0-Grid-Infrastructure-by-Alex-Gorbachev-Pythian-57-320.jpg)
UTHOC2 - Under The Hood of Oracle Clusterware 2.0 - Grid Infrastructure by Alex Gorbachev Pythian
- 1. Under the Hood of Oracle Clusterware 2.0: Grid Infrastructure Alex Gorbachev 29 August, 2012
- 2. Agenda • Place of Grid Infrastructure in Oracle RAC • Node membership and evictions • Clusterware architecture & startup sequence • Resources Management and troubleshooting 2 © 2009/2010 Pythian
- 3. Agenda Th High th e e le m ss or yo e yo Need to memorize u u ne u ed nd to ers m tan em d, or ize Low Shallow In-depth Understanding 2 © 2009/2010 Pythian
- 4. Single Instance Oracle Database APP Query/DML/DDL SERVER INSTANCE Memory (SGA, PGA) Processes (PMON, SMON, LGWR and etc. + multiple shadow processes) Read/write Datafiles Controlfiles redo logs flashback logs, change tracking and etc... Database 3 © 2012 Pythian
- 5. Single Instance Oracle Database APP SERVER INSTANCE Database 4 © 2012 Pythian
- 6. Oracle RAC Database APP SERVER 1 INSTANCE 1 Database 5 © 2012 Pythian
- 7. Oracle RAC Database APP SERVER 1 SERVER 2 INSTANCE 1 INSTANCE 2 Database 5 © 2012 Pythian
- 8. Oracle RAC Database APP SERVER 1 SERVER 2 SERVER 3 INSTANCE 1 INSTANCE 2 INSTANCE 3 Database 5 © 2012 Pythian
- 9. Oracle RAC Database APP SERVER 1 SERVER 2 SERVER 3 INSTANCE 1 INSTANCE 2 INSTANCE 3 Database 5 © 2012 Pythian
- 10. Oracle RAC Database APP SERVER 1 SERVER 2 SERVER 3 INSTANCE 1 INSTANCE 2 INSTANCE 3 Database 5 © 2012 Pythian
- 11. RAC looks simple. Eh? 6 © 2012 Pythian
- 12. Role of Grid Infrastructure OS OS OS VIP VIP VIP Listener Listener Listener Service Service Service Instance Instance Instance ASM ASM ASM Grid Infrastr. Grid Infrastr. Grid Infrastr. interconnect storage access OCR Voting disk Shared storage 7 © 2009/2010 Pythian
- 13. Role of Grid Infrastructure OS OS OS VIP VIP VIP Listener Listener Listener Service Service Service Instance Instance Instance ASM ASM ASM Grid Infrastr. Grid Infrastr. Grid Infrastr. interconnect storage access OCR Voting disk Shared storage 7 © 2009/2010 Pythian
- 14. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD 8 © 2009/2010 Pythian
- 15. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD 8 © 2009/2010 Pythian
- 16. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD 8 © 2009/2010 Pythian
- 17. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD Voting disk 9 © 2009/2010 Pythian
- 18. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD Voting disk 9 © 2009/2010 Pythian
- 19. OS OS Shoot Clusterware Clusterware The Other Node CSSD CSSD interconnect In OPROCD OPROCD The Head Voting disk 9 © 2009/2010 Pythian
- 20. OS OS Clusterware Clusterware VIP RACG EVMD CRSD CSSD CSSD interconnect OPROCD OPROCD Voting disk 10 © 2009/2010 Pythian
- 21. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD Voting disk 11 © 2009/2010 Pythian
- 22. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD Voting disk 11 © 2009/2010 Pythian
- 23. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD Voting disk 11 © 2009/2010 Pythian
- 24. OS Clusterware CSSD CSSD interconnect OPROCD Voting disk 11 © 2009/2010 Pythian
- 25. OS Clusterware Ask The OtherCSSD CSSD Node interconnect OPROCD To Reboot Voting Itself (c) known quote disk 11 © 2009/2010 Pythian
- 26. 11gR2 Grid Infrastructure: CSSD attempts graceful shutdown 12 © 2009/2010 Pythian
- 27. OS OS Clusterware Clusterware CS SD CSSD interconnect OPROCD OPROCD Voting disk 13 © 2009/2010 Pythian
- 28. OS OS Clusterware Clusterware CSSD Monitor/Agent CS SD CSSD interconnect OPROCD OPROCD Voting disk 13 © 2009/2010 Pythian
- 29. OS Clusterware CSSD Monitor/Agent CSSD interconnect OPROCD Voting disk 13 © 2009/2010 Pythian
- 30. OS Clusterware CSSD CSSD interconnect OPROCD Voting disk 14 © 2009/2010 Pythian
- 31. OS Clusterware CSSD CSSD interconnect OPROCD Voting disk 14 © 2009/2010 Pythian
- 32. OS Clusterware CSSD CSSD interconnect OPROCD Voting disk 14 © 2009/2010 Pythian
- 33. OS Clusterware CSSD CSSD interconnect OPROCD/CSSD Mon OPROCD Voting disk 14 © 2009/2010 Pythian
- 34. OS Clusterware CSSD interconnect OPROCD/CSSD Mon OPROCD Voting disk 14 © 2009/2010 Pythian
- 35. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD Voting disk 15 © 2009/2010 Pythian
- 36. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD Voting disk 15 © 2009/2010 Pythian
- 37. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD Voting disk 15 © 2009/2010 Pythian
- 38. OS Clusterware CSSD CSSD interconnect OPROCD Voting disk 15 © 2009/2010 Pythian
- 39. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD Voting disk 16 © 2009/2010 Pythian
- 40. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD 16 © 2009/2010 Pythian
- 41. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD 16 © 2009/2010 Pythian
- 42. CSSD CSSD interconnect 16 © 2009/2010 Pythian
- 43. 11gR2 Grid Infrastructure: CSSD attempts graceful shutdown 17 © 2009/2010 Pythian
- 44. OS OS Clusterware Clusterware Instance Instance LMON CSSD CSSD interconnect OPROCD OPROCD 18 © 2009/2010 Pythian
- 45. OS OS Clusterware Clusterware Instance Instance LMON member kill CSSD CSSD interconnect OPROCD OPROCD 18 © 2009/2010 Pythian
- 46. OS OS Clusterware Clusterware Instance Instance LMON member kill CSSD CSSD interconnect OPROCD OPROCD 18 © 2009/2010 Pythian
- 47. OS OS Clusterware Clusterware Instance Instance LMON member kill CSSD CSSD interconnect OPROCD OPROCD 18 © 2009/2010 Pythian
- 48. OS OS Clusterware Clusterware Instance Instance LMON member kill CSSD CSSD interconnect OPROCD OPROCD 18 © 2009/2010 Pythian
- 49. OS OS Clusterware Clusterware Instance Instance LMON member kill CSSD CSSD interconnect OPROCD OPROCD Eviction by escalation of a member kill 18 © 2009/2010 Pythian
- 50. OS Clusterware Instance LMON member kill CSSD interconnect OPROCD Eviction by escalation of a member kill 18 © 2009/2010 Pythian
- 51. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD Voting disk 19 © 2009/2010 Pythian
- 52. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD 11gR2 Intelligent Platform Voting Management disk Interface 19 © 2009/2010 Pythian
- 53. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD 11gR2 Intelligent Platform Voting Management disk Interface 19 © 2009/2010 Pythian
- 54. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD 11gR2 Intelligent Platform Voting Management disk Interface 19 © 2009/2010 Pythian
- 55. OS OS Clusterware Clusterware CSSD CSSD interconnect OPROCD OPROCD Voting Exadata Fencing disk 20 © 2009/2010 Pythian
- 56. Grid Infrastructure Startup 10g / 11gR1 11gR2 • Linux & UNIX inittab • Linux & UNIX inittab init.cssd init.ohasd run init.evmd init.crsd • Linux & UNIX init.d • Linux & UNIX init.d init.crs ohasd start • Windows Services • Windows Services 21 © 2009/2010 Pythian
- 57. Startup in Linux & Unix [root@cheese2 ~]# ps -fe | grep 'init.' | grep -v grep root 4283 1 0 02:52 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run [root@cheese2 ~]# tail -1 /etc/inittab h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null [root@cheese2 ~]# ls -l /etc/rc3.d/*ohasd* lrwxrwxrwx 1 root root 17 Sep 15 02:02 /etc/rc3.d/K15ohasd -> /etc/init.d/ohasd lrwxrwxrwx 1 root root 17 Sep 15 02:02 /etc/rc3.d/S96ohasd -> /etc/init.d/ohasd 22 © 2009/2010 Pythian
- 58. Pre 11gR2 Clusterware Startup /etc/oracle/scls_scr/{host}/root/cssrun /etc/oracle/scls_scr/{host}/root/crsstart • enable • disable init.crs start init.cssd autostart init.cssd oprodc oprocd init.cssd oclsomon oclsomon.bin init.cssd oclsvmon oclsvmon.bin init.cssd daemon ocssd.bin init.cssd fatal evmd.bin init.evmd run init.crsd run crsd.bin t 23 © 2009/2010 Pythian
- 59. Grid Infrastructure Startup t 24 © 2009/2010 Pythian
- 60. Grid Infrastructure Startup init.ohasd run t 24 © 2009/2010 Pythian
- 61. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun init.ohasd run t 24 © 2009/2010 Pythian
- 62. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun init.ohasd run t 24 © 2009/2010 Pythian
- 63. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun /etc/init.d/ohasd init.ohasd run t 24 © 2009/2010 Pythian
- 64. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun /etc/init.d/ohasd /etc/oracle/scls_scr/{host}/root/ohasdstr • enable • disable init.ohasd run t 24 © 2009/2010 Pythian
- 65. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun /etc/init.d/ohasd /etc/oracle/scls_scr/{host}/root/ohasdstr • enable • disable init.ohasd run t 24 © 2009/2010 Pythian
- 66. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun init.ohasd run t 24 © 2009/2010 Pythian
- 67. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun OHAS init.ohasd run t 24 © 2009/2010 Pythian
- 68. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun Ora Agent OHAS init.ohasd run t 24 © 2009/2010 Pythian
- 69. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun Ora Agent OHAS Root Agent init.ohasd run t 24 © 2009/2010 Pythian
- 70. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun Ora Agent OHAS Root Agent CSSD Agent init.ohasd run t 24 © 2009/2010 Pythian
- 71. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun Ora Agent OHAS Root Agent CSSD Agent init.ohasd run CSSD Monitor t 24 © 2009/2010 Pythian
- 72. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun Ora Agent OHAS Root Agent CSSD Agent CSS init.ohasd run CSSD Monitor t 24 © 2009/2010 Pythian
- 73. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun Ora Agent ACFS Drivers CTSS Disk Monitor OHAS Root Agent CRS CSSD Agent CSS init.ohasd run CSSD Monitor t 24 © 2009/2010 Pythian
- 74. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun ASM EVM GPnP GIPC Ora Agent MDNS ACFS Drivers CTSS Disk Monitor OHAS Root Agent CRS CSSD Agent CSS init.ohasd run CSSD Monitor t 24 © 2009/2010 Pythian
- 75. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun ASM EVM GPnP CRS Root Agent GIPC •VIP •SCAN IP Ora Agent MDNS •Network •GNS •ACFS Registry ACFS Drivers CTSS Disk Monitor OHAS Root Agent CRS CSSD Agent CSS init.ohasd run CSSD Monitor t 24 © 2009/2010 Pythian
- 76. Grid Infrastructure Startup /etc/oracle/scls_scr/{host}/root/ohasdrun ASM EVM GPnP CRS Root Agent GIPC •VIP •SCAN IP Ora Agent MDNS •Network •GNS •ACFS Registry ACFS Drivers CTSS CRS Ora Agent Disk Monitor •Database •Instance OHAS Root Agent CRS •Listener •Services •Diskgroups CSSD Agent CSS •ONS •eONS •SCAN Listener init.ohasd run CSSD Monitor t 24 © 2009/2010 Pythian
- 77. Grid Infrastructure Startup (static slide) ASM EVM GPnP CRS Root Agent GIPC •VIP •SCAN IP Ora Agent MDNS •Network •GNS •ACFS Registry ACFS Drivers CTSS CRS Ora Agent Disk Monitor •Database •Instance OHAS Root Agent CRS •Listener •Services •Diskgroups CSSD Agent CSS •ONS •eONS •SCAN Listener init.ohasd run CSSD Monitor t 25 © 2009/2010 Pythian
- 78. Grid Infrastructure Log Files $GRID_HOME/log/{hostname}/ • alert<host>.log • ctssd • ohasd • diskmon • crsd • gipcd • cssd • gnsd • agent/ohasd/oraagent_oracle • gpnpd • agent/ohasd/oracssdagent_root • mdnsd • agent/ohasd/oracssdmonitor_root • racg • agent/ohasd/orarootagent_root • agent/crsd/oraagent_oracle • agent/crsd/orarootagent_root 26 © 2009/2010 Pythian
- 79. Oracle Cluster Registry • Repository for all shared configuration data • Except OCR location itself • OCR is accessed mostly read-only • Every component reads OCR • OCR is written only by CRS • only from a single OCR master node • 11gR2 - Oracle Local Registry (OLR) • managed by ohasd 27 © 2009/2010 Pythian
- 80. DEMO Interconnect Failure • Simulate with “ifconfig eth1 down” on node 2 • Both nodes notice the loss • Racing to evict each other • from voting disk => 2 equal sub-clusters (cohorts) • survives the one with the lowest leader # • leader is the node with lowest # in sub-cluster • Winner evicts another node • Setting kill-block in voting disk • 11gR2 new feature: CSSD does clean restart 28 © 2009/2010 Pythian
- 81. DEMO Voting Disk Failure • Simulate with storage interface down or on NFS server • CSSD detects voting disk IO stale • disktimeout setting - 200 seconds default • CSSD starts eviction • 11gR2 new feature: CSSD does clean restart 29 © 2009/2010 Pythian
- 82. DEMO CSSD is not healthy • Simulate using kill -STOP <cssd.bin pid> (and try kill -9) • Another node observes NHB loss • After misscount seconds => attempt eviction • but CSSD is frozen and can’t commit suicide • CSSD Monitor detects CSSD timeout • Commit suicide 30 © 2009/2010 Pythian
- 83. Homework Host sick - CPU stalled • Used to simulate by pausing OPROCD • kill -STOP <oprocd pid> • sleep 1 or 2 • kill -CONT <oprocd pid> • OPROCD is now a thread in CSSD Monitor • kill -STOP {cssdmonitor.bin} ; sleep 1 ; kill -CONT {cssdmonitor.bin} • Doesn’t produce any visible results! Is there still OPROCD? 31 © 2009/2010 Pythian
- 84. DEMO Startup troubleshooting • Break before starting up • Interconnect, voting disk, Grid Home missing • Check processes using “ps -fe | grep init” • Check syslog (/var/log/messages) • boot sequence • Clusterware log files • if *.bin processes are running already • crsctl check crs • crsctl status resource -t -init 32 © 2009/2010 Pythian
- 85. 11gR2 Cluster Resources • Introduced resource type • APPLICATION was the only supported resource type in 11gR1 • Resource types using resource agents • Cluster and local resources • Sophisticated resources dependencies (stop & start) • hard • weak • attraction • pullup • dispersion 33 © 2009/2010 Pythian
- 86. Grid Infrastructure Processes • ora.cluster_interconnect.haip ASM CRS Managed Resources EVM OHAS Internal Resources GPnP CRS Root Agent GIPC • VIP • SCAN IP Ora Agent MDNS • Network • GNS • ACFS Registry ACFS Drivers CTSS CRS Ora Agent Disk Monitor • Database • Instance OHAS Root Agent CRS • Listener • Services • Diskgroups CSSD Agent CSS • ONS • eONS • SCAN Listener init.ohasd run CSSD Monitor t 34 © 2009/2010 Pythian
- 87. Troubleshooting Something Down • OHASD up? • OHASD Agents up? • Internal resources up? • CRSD Agents up? • Managed resources up? • Listener & VIP • Database & ASM instance • Services • Have the nodes rebooted? • Have resources re-started? 35 © 2009/2010 Pythian
- 88. 11gR2 Grid Infrastructure References • Oracle Clusterware Administration and Deployment Guide • MOS 1053147.1 • 11gR2 Clusterware and Grid Home - What You Need to Know • MOS 1050908.1 • How to Troubleshoot Grid Infrastructure Startup Issues • MOS 1053970.1 • Troubleshooting 11.2 Grid Infastructure Installation Root.sh Issues • MOS 1050693.1 • Troubleshooting 11.2 Clusterware Node Evictions (Reboots) • MOS 942166.1 • How to Proceed from Failed 11gR2 Grid Infrastructure Installation 36 © 2009/2010 Pythian