z/VM Performance Analysis
Download as PPT, PDF1 like5,888 views
The document provides guidelines and commands for analyzing z/VM performance. It discusses processor, storage, paging, and server machine guidelines. Key performance metrics like CPU utilization, response time, and throughput are defined. Commands for monitoring load, queues, paging, storage, and more are listed. REORDMON is introduced as a tool for diagnosing reorder processing overhead.
z/VM Performance Analysis
- 1. 1 z/VM Performance Analysis Lívio Sousa - [email protected] IBM zEnterprise Client Technical Specialist
- 2. 2 Overview • Guidelines • Commands • *MONITOR • Performance Toolkit • Omegamon XE
- 3. 3 Definition of Performance • Performance definitions: – Response time – Batch elapsed time – Throughput – Resource consumed per unit of work done – Utilization – Users supported – Phone ringing – Consistency • All of the above
- 4. 4 Performance Guidelines • Processor • Storage • Paging • Minidisk cache • Server machines
- 5. 5 Processor Guidelines • Dedicated processors - mostly political – Absolute share can be almost as effective – Gets wait state assist and 500 ms minor time slice – Perhaps not a good idea if you are CPU-constrained – A virtual machine should have all dedicated or all shared processors • Share settings – Use absolute if you can judge percent of resources required – Use relative if difficult to judge and if lower share as system load increases is acceptable – Be aware that share value is split by vCPUs – Do not use LIMITHARD settings unnecessarily • Masks looping users • More scheduler overhead • Use the right number of virtual processors for the guest's workload • Don’t share all available IFLs to all LPARs – Suspend Time can be high
- 6. 6 Storage Guidelines Virtual-to-real ratio should be <= 3:1 or make sure paging system is robust – To avoid any performance impact for production workloads, you may need to keep ratio to 1:1 – See also http://www.vm.ibm.com/perf/tips/memory.html – VIR2REAL EXEC (Bruce Hayden) http://www.vm.ibm.com/download/packages/descript.cgi?VIR2REAL Define some processor storage as expanded storage to provide paging hierarchy – For more background, see http://www.vm.ibm.com/perf/tips/storconf.html Size guests appropriately – Avoiding over provisioning – Do not put them in a high guest paging position – Right-sized usually means "just barely swapping" Exploit shared memory where possible – IPL your Linux guests from a segment – Use the Linux XIP (execute-in-place) file system Total Virtual storage (all logged on userids): 388308 MB (379.2 GB) Usable real storage (pageable) for this system: 202927 MB (198.2 GB) Total LPAR Real storage: 204800 MB (200.0 GB) Expanded storage usable for paging: 25600 MB ( 25.0 GB) Total Virtual disk (VDISK) space defined: 50176 MB ( 49.0 GB) Average Virtual disk size: 512 MB Virtual + VDISK to Real storage ratio: 2.2 : 1
- 7. 7 Paging Guidelines • DASD paging allocations less than or equal to 50% – QUERY ALLOC PAGE • Watch blocks read per paging request (keep >10) – Long block runs make paging I/O efficient • Multiple volumes and multiple paths – Remember, one I/O per real device at a time – Use lots of little volumes rather than a few big volumes – Pay attention in Response Time and Wait Queues • Do not mix sizes of paging DASD – Use all -3s, or all -9s, or whatever • Paging to FCP SCSI (EDEVICES) may offer higher paging bandwidth with higher processor requirements – See also http://www.vm.ibm.com/perf/tips/prgpage.html
- 8. 88 Reorder Processing - Background • Page reorder is the process in z/VM of managing user frame owned lists as input to demand scan processing. – It includes resetting the HW reference bit. – Serializes the virtual machine (all virtual processors). – In all releases of z/VM • It is done periodically on a virtual machine basis. • The cost of reorder is proportional to the number of resident frames for the virtual machine. – Roughly 130 ms/GB resident – Delays of ~1 second for guest having 8 GB resident – This can vary for different reasons +/- 40%
- 9. 9 9 Reorder Processing - Diagnosing Performance Toolkit – Check resident page fields (“R<2GB” & “R>2GB”) on FCX113 UPAGE report • Remember, Reorder works against the resident pages, not total virtual machine size. – Check Console Function Mode Wait (“%CFW”) on FCX114 USTAT report • A virtual machine may be brought through console function mode to serialize Reorder. There are other ways to serialize for Reorder and there are other reasons that for CFW, so this is not conclusive. REORDMON – Available from the VM Download Page http://www.vm.ibm.com/download/packages/ – Works against raw MONWRITE data for all monitored virtual machines – Works in real time for a specific virtual machine – Provides how often Reorder processing occurs in each monitor interval
- 10. 10 10 REORDMON Example Num. of Average Average Userid Reorders Rsdnt(MB) Ref'd(MB) Reorder Times -------- -------- --------- --------- ------------------- LINUX002 2 18352 13356 13:29:05 14:15:05 LINUX001 1 22444 6966 13:44:05 LINUX005 1 14275 5374 13:56:05 LINUX003 2 21408 13660 13:43:05 14:10:05 LINUX007 1 12238 5961 13:51:05 LINUX006 1 9686 4359 13:31:05 LINUX004 1 21410 11886 14:18:05
- 11. 11 11 Reorder Processing - Mitigations • Try to keep the virtual machine as small as possible. • Virtual machines with multiple applications may need to be split into multiple virtual machines with fewer applications. • See http://www.vm.ibm.com/perf/tips/reorder.html for more details. • Apply APAR VM64774 if necessary: – SET and QUERY commands, system wide settings – Corrects problem in earlier “patch” solution that inhibits paging of PGMBKs (Page Tables) for virtual machines where Reorder is set off. – z/VM 5.4.0 PTF UM33167 RSU 1003 – z/VM 6.1.0 PTF UM33169 RSU 1003
- 12. 12 Minidisk Cache Guidelines • In general, enable MDC for everything • Configure some real storage for MDC • Set maximum MDC limits – SET MDC STOR 0M 256M and SET MDC XSTOR 0M 0M • Disable MDC for – Write-mostly or read-once disks (logs, accounting, Linux swap) – Target volumes in backup scenarios • Better performer than Virtual Disk in Storage (VDISK) for read I/Os
- 13. 13 Server Machine Guidelines • Server Virtual Machine (SVM) • TCP/IP, RACFVM, etc. • QUICKDSP ON to avoid eligible list • Higher SHARE setting • Ensure performance data includes these virtual machines
- 14. 14 CP INDICATE Command • LOAD: shows total system load – Processors, XSTORE, paging, MDC, queue lengths, storage load – STORAGE value not very meaningful • USER EXP: more useful than plain USER • QUEUES EXP: great for scheduler problems and quick state sampling – Mostly useful for eligible list assessments • PAGING: lists users in page wait • I/O: lists users in I/O wait • ACTIVE: displays number of active users over given interval • Consider using MONITOR DATA instead for "serious" examinations
- 15. 15 CP INDICATE LOAD Example INDICATE LOAD AVGPROC-088% 03 XSTORE-000000/SEC MIGRATE-0000/SEC MDC READS-000035/SEC WRITES-000001/SEC HIT RATIO-099% PAGING-0023/SEC STEAL-000% Q0-00007(00000) DORMANT-00410 Q1-00000(00000) E1-00000(00000) Q2-00001(00000) EXPAN-002 E2-00000(00000) Q3-00013(00000) EXPAN-002 E3-00000(00000) PROC 0000-087% PROC 0001-088% PROC 0002-089% LIMITED-00000
- 16. 16 Selected CP QUERY Commands USERS: number and type of users on system SRM: scheduler/dispatcher settings (LDUBUF, etc.) SHARE: type and intensity of system share FRAMES: real storage allocation PATHS: physical paths to device and status ALLOC MAP: DASD allocation ALLOC PAGE: how full your paging space is XSTORE: assignment of expanded storage MONITOR: current monitor settings MDC: MDC usage VDISK: virtual disk in storage usage SXSPAGES: System Execution Space
- 17. 17 5,000 Foot View CP Control Blocks Application Data VM Events *MONITOR System Service MONDCSS Segment MONWRITE Utility Performance Toolkit Raw Monwrite History Files TCP/IP Network 3270 Browser VMRM
- 18. 18
- 19. 19 Processor REPORT NAME REPORT CODE COMMAND CPU Load and Transactions FCX100 CPU LPAR Load FCX126 LPAR Processor Log FCX144 PROCLOG LPAR Load Log FCX202 LPARLOG User Wait States FCX114 USTAT System Summary FCX225 SYMSUMLG
- 20. 20 FCX126 Run 2011/09/20 18:00:56 Logical Partition Activity From 2011/09/13 09:19:15 To 2011/09/13 10:09:15 For 3000 Secs 00:50:00 Result of 13092011 Run __________________________________________________________________________________________ Processor type and model : 2817-401 Nr. of configured partitions: 6 Nr. of physical processors : 25 Partition Nr. Upid #Proc Weight Wait-C Cap %Load CPU %Busy %Ovhd %Susp %VMld %Logld Type LPAR1 1 00 24 100 NO NO 89.0 0 94.3 2.1 6.5 92.0 98.4 IFL 100 NO 1 93.4 2.4 7.7 90.8 98.3 IFL 100 NO 2 93.6 2.3 7.4 91.1 98.3 IFL 100 NO 3 93.6 2.4 7.5 91.1 98.4 IFL 100 NO 4 93.6 2.3 7.4 91.1 98.4 IFL 100 NO 5 93.5 2.3 7.5 91.0 98.3 IFL 100 NO 6 93.4 2.4 7.6 90.9 98.3 IFL 100 NO 7 93.2 2.4 7.7 90.6 98.1 IFL 100 NO 8 93.4 2.4 7.5 90.8 98.2 IFL 100 NO 9 93.2 2.4 7.7 90.7 98.2 IFL 100 NO 10 93.1 2.5 7.8 90.4 98.0 IFL 100 NO 11 93.2 2.4 7.7 90.6 98.0 IFL 100 NO 12 93.4 2.4 7.5 90.8 98.1 IFL 100 NO 13 93.3 2.3 7.5 90.8 98.1 IFL 100 NO 14 93.3 2.4 7.5 90.7 98.1 IFL 100 NO 15 93.2 2.5 7.6 90.5 97.9 IFL 100 NO 16 91.1 2.9 9.0 88.0 96.6 IFL 100 NO 17 91.3 2.8 8.8 88.2 96.7 IFL 100 NO 18 91.4 2.9 8.9 88.3 96.8 IFL 100 NO 19 91.5 2.7 8.8 88.5 97.0 IFL 100 NO 20 91.7 2.8 8.7 88.6 97.1 IFL 100 NO 21 91.5 2.8 8.9 88.5 97.1 IFL
- 21. 21 FCX225 Run 2011/09/20 18:00:56 SYSSUMLG System Performance Summary by Time From 2011/09/13 09:19:15 To 2011/09/13 10:09:15 For 3000 Secs 00:50:00 Result of 13092011 Run _________________________________________________________________________________ <------- CPU --------> <Vec> <--Users--> <---I/O---> <Stg> <-Paging--> <--Ratio--> SSCH DASD Users <-Rate/s--> Interval Pct Cap- On- Pct Log- +RSCH Resp in PGIN+ Read+ End Time Busy T/V ture line Busy ged Activ /s msec Elist PGOUT Write >>Mean>> 90.0 1.10 .9293 24.0 .... 117 108 571.2 .4 .0 2610 1051 09:20:15 92.4 1.13 .9059 24.0 .... 117 108 523.0 .5 .0 1992 527.8 09:21:15 92.9 1.07 .9523 24.0 .... 117 108 399.2 .5 .0 1669 301.4 09:22:15 93.2 1.08 .9458 24.0 .... 117 108 557.4 .3 .0 2817 633.9 09:23:15 94.5 1.07 .9535 24.0 .... 117 108 590.3 .3 .0 1410 482.7 09:24:15 93.4 1.07 .9537 24.0 .... 117 108 649.5 .4 .0 2363 488.5 09:25:15 90.4 1.09 .9347 24.0 .... 117 108 684.7 .4 .0 2485 768.9 09:26:15 92.4 1.08 .9436 24.0 .... 117 108 666.8 .4 .0 2940 1215 09:27:15 90.9 1.09 .9344 24.0 .... 117 108 607.2 .4 .0 3179 726.7 09:28:15 92.2 1.08 .9469 24.0 .... 117 108 664.2 .5 .0 2179 896.0 09:29:17 90.8 1.10 .9318 24.0 .... 117 108 645.9 .6 .0 3404 804.5 09:30:16 89.5 1.19 .8579 24.0 .... 117 108 670.8 .7 .0 5402 3487 09:31:15 92.7 1.08 .9412 24.0 .... 117 108 588.7 .4 .0 3091 1807 09:32:15 91.2 1.09 .9421 24.0 .... 117 108 602.8 .3 .0 2635 1076 09:33:16 89.3 1.14 .9047 24.0 .... 117 108 255.2 .5 .0 3140 710.5 09:34:15 88.5 1.10 .9374 24.0 .... 117 108 205.2 .6 .0 2513 897.4 09:35:15 85.9 1.12 .9257 24.0 .... 117 108 320.4 .5 .0 3117 953.5 09:36:16 86.1 1.13 .9144 24.0 .... 117 108 213.5 .5 .0 3642 1144 09:37:16 83.0 1.14 .9090 24.0 .... 117 108 245.6 .5 .0 3414 2133
- 22. 22 REPORT NAME REPORT CODE COMMAND Auxiliary Storage Log FCX146 AUXLOG CP Owned Device FCX109 DEVICE CPOWNED User Page Data FCX113 UPAGE Shared Data Spaces FCX134 DSPACESH SXS Available Page Queues Mgnt FCX261 SXSAVAIL Mini Disk Storage FCX178 MDCSTOR Storage Utilization FCX103 STORAGE Available List Log FCX254 AVAILLOG Storage
- 23. 23 FCX109 Run 2011/05/31 17:44:26 DEVICE CPOWNED Load and Performance of CP Owned Disks From 2011/05/12 16:48:41 To 2011/05/12 17:31:41 For 2580 Secs 00:43:00 Result of 20110512 Run _______________________________________________________________________________ Page / SPOOL Allocation Summary PAGE slots available 25165k SPOOL slots available 3605928 PAGE slot utilization 25% SPOOL slot utilization 65% T-Disk cylinders avail. ....... DUMP slots available 0 T-Disk space utilization ...% DUMP slot utilization ..% . . . . . . . . . . _____ . .< Device Descr. -> <------------- Rate/s -------------> User Serv MLOAD Volume Area Area Used <--Page---> <--Spool--> SSCH Inter Queue Time Resp Addr Devtyp Serial Type Extent % P-Rds P-Wrt S-Rds S-Wrt Total +RSCH feres Lngth /Page Time EDF1 9336 ZDPAG1 PAGE 12583k 25 196.5 199.9 ... ... 396.4 .0 0 8.18 5.5 88.0 EDF2 9336 ZDPAG2 PAGE 12583k 24 194.2 206.1 ... ... 400.4 .0 0 7.23 6.0 58.4 4374 3390 610SP1 SPOOL 802880 61 .0 .0 .0 .0 .0 .1 0 0 .4 .4 4672 3390 610SP2 SPOOL 803060 68 .0 .0 .0 .0 .0 .0 0 0 1.0 1.0
- 24. 24 I/O REPORT NAME REPORT CODE COMMAND General I/O Device FCX108 DEVICE SCSI Device FCX249 SCSI DASD Performance Log FCX131 DEVCONF FICON Channel Load FCX215 INTERIM FCHANNEL General I/O Device Data Log FCX168 DEVLOG I/O Processor Log FCX232 IOPROCLG
- 25. 25 Studying MONWRITE Data • z/VM Performance Toolkit • Interactively – possible, but not so useful • PERFKIT BATCH command – pretty useful – Control files tell Perfkit which reports to produce – You can then inspect the reports by hand or programmatically • See z/VM Performance Toolkit Reference for information on how to use PERFKIT BATCH • PRFIT (Brian Wade) http://www.vm.ibm.com/download/packages/descript.cgi?PRFIT
- 26. 26 26 Some Notes on z/VM Limits • Sheer hardware: – z/VM 5.2: 24 engines, 128 GB real – z/VM 5.3: 32 engines, 256 GB real – zSeries: 65,000 I/O devices • Workloads we’ve run in test have included: – 54 engines – 440 GB real storage – 128 GB XSTORE – 240 1-GB Linux guests – 8 1-TB guests • Utilizations we routinely see in customer environments – 85% to 95% CPU utilization without worry – Tens of thousands of pages per second without worry • Our limits tend to have two distinct shapes – Performance drops off slowly with utilization (CPUs) – Performance drops off rapidly when wall is hit (storage) Performance Utilization Precipitous (e.g., storage) Gradual (e.g., CPUs)
- 27. 27 Some Final Thoughts • Define what is performance for your case • Collect data for a base line of good performance • Implement change management process • Make as few changes as possible at a time • Relieving one bottleneck will reveal another
- 28. 28 OBRIGADO! Informações de Contato: Livio Sousa IBM Tutóia – SP [email protected] +55 11 9 7203 6637