Geek Sync I Need for Speed: In-Memory Databases in Oracle and SQL Server
Download as pptx, pdf2 likes482 views
The document provides a comprehensive overview of in-memory databases in Oracle and SQL Server, highlighting key features, benefits, and limitations. It discusses Oracle's In-Memory Column Store (IMCS) and SQL Server's In-Memory OLTP, detailing their structures, performance advantages, and settings required for implementation. The document also includes benchmarking results and considerations for using these technologies effectively in enterprise environments.
Geek Sync I Need for Speed: In-Memory Databases in Oracle and SQL Server
- 1. In Memory Databases in SQL Server and Oracle May 4, 2016
- 2. Who Are We Bert Scalzo • Senior PM with IDERA • Author (10 books), Speaker, Oracle subject matter expert Vicky Harp • Strategist with IDERA • Developer, Speaker, SQL Server subject matter expert
- 3. Agenda • Introductions • What are In Memory Databases • In Memory Databases in Oracle • In Memory Databases in SQL Server • Wrap-Up
- 4. In Memory Databases • Databases or table structures which reside entirely or primarily in main memory • Able to make efficient use of the huge RAM storage available on modern systems • Able to utilize structures and algorithms which would be inefficient on disk • May or may not be fully ACID compliant • May be 100s of times faster
- 5. In Memory for Oracle
- 6. In Memory for Oracle • In Memory Column Store (IMCS) • New Feature Oracle 12c 2nd release – 12.1.0.2.0 – July 214 – not in 12.1.0.1.0 – 2013 • The IMCS is a new static pool in the SGA. • Data in IMCS does not reside in traditional row format, instead, in a columnar format. • Each column is stored as a separate structure. • IMCS does not replace the buffer cache, but it acts as a supplement so data can be stored in memory in both row & columnar formats. • To enable IMCS, INMEMORY_SIZE init parameter must = a non-zero value. • The IMCS is included with the Oracle Enterprise Edition • IMCS is an option (i.e. not free, costs $$$ like partitioning).
- 7. IMCS Pros & Cons IMCS can apply to: • Column • Table • Materialized View • Tablespace • Partition IMCS is good for: • A query that scans a large number of rows and applies filters that use operators such as: =, <, >, and IN • A query that selects a small number of columns from a table or materialized view with a large number of columns, such as a query that selects five columns from a table with 100 columns (e.g. DW Dimension) • A query that joins a small table to a large table (e.g. DW Dimension) • A query that aggregates data (e.g. DW fact)
- 8. IMCS Pros & Cons Availability: • 12c 12.1.0.2 • Enterprise Edition • EE Option $$$ (see next slide) IMCS is not good for: • IM is NOT good for: • Queries with complex predicates • Queries that selects lots of columns • Queries that return lots of rows • Queries with multiple large table joins • Objects owned by SYS and stored in SYSTEM or SYSAUX tablespaces
- 10. Oracle SGA As Oracle Database populates data in the IM column store and converts this data to columnar format, the columnar version of the table gradually becomes available. If a query requires data that has not yet been populated in the IM column store, then the database reads the requested data from the buffer cache or disk while background processes asynchronously populate the missing data in the IM column store.
- 15. SGA Memory Considerations 15 Green star merely indicates What I chose for my tests
- 17. Loading Options When a database is restarted, all of the data for database objects with a priority level other than NONE is populated in the IM column store during startup.
- 18. Test Approach The TPC Benchmark™ H (TPC-H) is a decision support benchmark. It consists of a suite of business-oriented ad- hoc queries and concurrent data modifications. The queries and the data populating the database were chosen to have broad industry-wide relevance, while maintaining a sufficient degree of ease of implementation. This benchmark illustrates decision support systems that do the following: – Examine large volumes of data; – Execute queries with a high degree of complexity; – Give answers to critical business questions. http://www.tpc.org/tpc_documents_current_versions/pdf/tpch2.17.1.pdf
- 19. Example Queries National Market Share Query (Q8) Volume Shipping Query (Q7)
- 20. Test Results Test # Feature Debut Configuration Parameters Run Time (Mins:Secs) Compress Factor % Run Time Reduction 1 SGA_TARGET = 4G *** Nothing Else–Baseline *** 45:52 2 11g SGA_TARGET = 4G DB_FLASH_CACHE_SIZE = 16G DB_FLASH_CACHE_FILE = '/flash/flash01.dbf' 23:19 49% 3 SGA_TARGET = 20G *** Nothing Else – All memory buffer cache *** 19:50 57% 4 12c SGA_TARGET = 20G DB_BIG_TABLE_CACHE_PERCENT_TARGET = 80 PARALLEL_DEGREE_POLICY=auto ORA-600 5 12c SGA_TARGET = 20G INMEMORY_SIZE = 16G INMEMORY (default) PRIORITY CRITICAL (restart instance & wait for object to load) 05:01 3.63X 89% 6 12c SGA_TARGET = 20G INMEMORY_SIZE = 16G INMEMORY MEMCOMPRESS FOR QUERY HIGH PRIORITY CRITICAL (restart instance & wait for object to load) 04:56 4.68X 89%
- 21. Script #1 -- im_tabs.sql set linesize 256 set pagesize 999 set verify off col OBJECT format a30 SELECT owner||'.'||table_name OBJECT, inmemory INMEMORY, inmemory_priority PRIORITY, inmemory_distribute DISTRIBUTE, inmemory_compression COMPRESSION, inmemory_duplicate DUPLICATE FROM all_tables where owner like upper('%&1%') ORDER BY inmemory, owner||'.'||table_name; SQL> @im_tabs BMF1 OBJECT INMEMORY PRIORITY DISTRIBUTE COMPRESSION DUPLICATE ------------------------------ -------- -------- --------------- ----------------- ------------ BMF1.H_CUSTOMER ENABLED CRITICAL AUTO FOR QUERY LOW NO DUPLICATE BMF1.H_LINEITEM ENABLED CRITICAL AUTO FOR QUERY LOW NO DUPLICATE BMF1.H_NATION ENABLED CRITICAL AUTO FOR QUERY LOW NO DUPLICATE BMF1.H_ORDER ENABLED CRITICAL AUTO FOR QUERY LOW NO DUPLICATE BMF1.H_PART ENABLED CRITICAL AUTO FOR QUERY LOW NO DUPLICATE BMF1.H_PARTSUPP ENABLED CRITICAL AUTO FOR QUERY LOW NO DUPLICATE BMF1.H_REGION ENABLED CRITICAL AUTO FOR QUERY LOW NO DUPLICATE BMF1.H_SUPPLIER ENABLED CRITICAL AUTO FOR QUERY LOW NO DUPLICATE First, we need to know which tables for our test schema (BMF1) have been “in-memory” enabled and with which options.
- 22. Script #2 -- im_segs.sql set linesize 256 set pagesize 999 set verify off col owner format a20 col segment_name format a30 select owner, segment_name, populate_status from v$im_segments; SQL> @im_segs OWNER SEGMENT_NAME POP STATU -------------------- ------------------------------ --------- BMF1 H_LINEITEM STARTED BMF1 H_SUPPLIER COMPLETED BMF1 H_PARTSUPP COMPLETED BMF1 H_ORDER COMPLETED BMF1 H_PART COMPLETED BMF1 H_CUSTOMER COMPLETED Second, we need to know when our “in-memory” enabled tables have been successfully loaded. In my tests, I waited until all such enabled tables were COMPLETED to more fully leverage the in- memory feature.
- 23. Script #3 -- im_pct.sql set linesize 256 set pagesize 999 set verify off select segment_name,ROUND(SUM(BYTES)/1024/1024/1024,2) "ORIG GB", ROUND(SUM(INMEMORY_SIZE)/1024/1024/1024,2) "IN-MEM GB", ROUND(SUM(BYTES-BYTES_NOT_POPULATED)*100/SUM(BYTES),2) "% IN_MEM", ROUND(SUM(BYTES-BYTES_NOT_POPULATED)/SUM(INMEMORY_SIZE),2) "COMP RATIO" from V$IM_SEGMENTS group by owner,segment_name order by SUM(bytes) desc; SQL> @im_pct -- Try #1 - INMEMORY (default) SEGMENT_NAME ORIG GB IN-MEM GB % IN_MEM COMP RATIO ------------------------------ ---------- ---------- ---------- ---------- H_LINEITEM 8.42 2.32 100 3.63 H_ORDER 1.75 .89 100 1.97 H_PARTSUPP .6 .35 100 1.71 H_PART .29 .13 100 2.29 H_CUSTOMER .22 .19 100 1.19 H_SUPPLIER .01 .02 100 .91 SQL> @im_pct -- Try #2 - INMEMORY MEMCOMPRESS FOR QUERY HIGH SEGMENT_NAME ORIG GB IN-MEM GB % IN_MEM COMP RATIO ------------------------------ ---------- ---------- ---------- ---------- H_LINEITEM 8.42 1.77 98.17 4.68 H_ORDER 1.75 .63 100 2.78 H_PARTSUPP .6 .32 100 1.86 H_PART .29 .09 100 3.16 H_CUSTOMER .22 .15 100 1.52 H_SUPPLIER .01 .01 100 1.31 wait Third, we need to know how well our “in- memory” enabled tables compressed to confirm the Oracle statement that the in-memory feature does not add much overhead (20%).
- 24. In Memory for SQL Server
- 25. In Memory Databases in SQL Server • First appearance in SQL 2014 “Hekaton” • Enterprise Edition feature • Increased functionality in SQL 2016 • Called by Many Names – In-Memory OLTP – Memory Optimized Tables – Hekaton (or just HK)
- 26. SQL Server In-Memory OLTP • A distinct memory-optimized engine within SQL Server • Distinct storage and memory structures from the rest of SQL Server • Completely lock-less design eliminates blocking • ACID compliant with optional durability • Interacts with traditional TSQL or new natively-compiled stored procedures
- 27. SQL Server In-Memory Concepts • Memory Optimized Tables • Durability Options • T-SQL Interop • Natively Compiled Stored Procedures • Memory Optimized Table Types
- 28. Memory Optimized Tables • Created on a MEMORY_OPTIMIZED_DATA filegroup • Completely distinct physical structure than on-disk tables • Tables are compiled to a DLL on creation • Data and indexes fully resident in RAM • Fully optimistic (lockless) concurrency
- 29. Memory Optimized Table Structure • Data is written to and read from hash buckets in memory • Table data is persisted to the transaction log and periodically flushed to disk • Indexes are maintained only in memory and never persisted
- 30. Memory Optimized Table Limitations • Long list of miscellaneous unsupported features and commands – Computed columns, clustered indexes, sparse columns, DDL triggers, full text indexing, truncate, data compression, dbcc checkdb • Refer to documentation
- 31. Durability Options • SCHEMA_AND_DATA – Changes are logged in transaction log – Table structure and data are persisted on disk – ACID compliant • SCHEMA_ONLY – Changes are not logged – Table structure is persisted – Data is not persisted – Atomic, Consistent, and Isolated but not Durable
- 32. T-SQL Interop • Access Memory-Optimized tables directly using regular T-SQL • Allows “cross-container” access • Most (but not all) T-SQL language constructs are supported • Only certain isolation levels supported – Snapshot (with caveats) – Repeatable Read – Serializable
- 33. Natively Compiled Stored Procedures • Stored procedures written in T-SQL and compiled directly to DLLs • Do not incur compilation overhead • Much, much faster • Many unsupported T-SQL features – LIKE, CONTAINS, PIVOT, EXCEPT, common table expressions, cursors, select into, temp tables, etc
- 34. Memory Optimized Table Types • May be used as a temporary table but without making use of tempdb • May be desirable for temporary objects which can reside entirely in memory
- 35. Wrap Up
- 36. Wrap Up • Oracle (key points) – Point 1 – Point 2 – Point 3 • SQL Server Memory-Optimized Tables – Distinct engine and structure with lockless design – Optional table durability – Optional native compiled stored procedures
- 37. Thank you! • Bert Scalzo – [email protected] • Vicky Harp – [email protected] • http://community.idera.com
Editor's Notes
- #2: Welcome can be done by Bert or Vicky. Recommend Bert to move smoothly to next slide.
- #3: Bert to introduce himself Vicky to introduce herself
- #4: Bert or Vicky to go over agenda. Recommend Vicky as continuation of previous slide.
- #5: Bert AND Vicky go over this slide, extemporaneously Recommendation: Vicky to read bullet points, add a little color, converse (like 1 line) with Bert on each. Sample script: Vicky: “Databases which reside in main memory” continues to “That is to say, rather than having all your data on disk and then read into memory as it’s being accessed, in-memory objects reside primarily in memory and if they’re written to disk at all it’s only for the purposes of durability through a server reboot.” Bert: “Yeah, that’s a complete inversion of the way the enterprise RDBMS systems have traditionally worked. <etc>” Vicky: “These in-memory databases are able to make more efficient use of the huge RAM storage…” continuing to “Until recently the limitations of RAM on even large servers made this sort of feature impractical because any database which could fit entirely in memory was also small enough that the overhead of disk activities just wasn’t very meaningful. But now that we have really huge servers that’s changing.” Bert: (whatever) (and so forth) Let’s be sure to review what ACID stands for at the appropriate moment (ending with Bert transitioning to his slides)
- #37: This may need to be 2 slides, one for each platform