Java MySQL Connector & Connection Pool Features & Optimization
10 likes6,691 views
The document discusses the optimization and features of Java MySQL connectors and connection pools, detailing configurations, prepared statements, and performance comparisons between MySQL Connector/J and MariaDB Java Client. It emphasizes the importance of connection pool management, including issues like lingering transactions, pool sizing, and failover strategies. Various connection pool implementations are also mentioned, along with recommendations for best practices to enhance performance.
Java MySQL Connector & Connection Pool Features & Optimization
- 1. Java MySQL Connector & Connection Pool Features & Optimization Kenny Gryp <[email protected]> April 14, 2015 @gryp
- 2. DISCLAIMER Please excuse me for not being a Java developer 2
- 3. What I Don’t Like • Brussels Sprouts • Taxes • Calories • Java(’s chattiness) 3
- 5. MYSQL CONNECTORS CONNECTION POOLS Connectors Configuring Connector Creating A Database Connection Prepared Statements Example Transaction 5
- 6. MySQL Connector/J & MariaDB Java Client • MySQL Connector/J – Oracle – 5.1.35 Latest – Compatible with • MySQL • Percona Server • MariaDB 6
- 7. MySQL Connector/J & MariaDB Java Client • MariaDB Java Client • MariaDB • Fork from Drizzle Connector • Latest 1.1.8 • Compatible with – MySQL – Percona Server – MariaDB 7
- 8. MySQL Connector/J Features • Enterprise Plugin: Query Analyzer • MySQL Fabric Integration • Load Balancing • Failover • Replication 8
- 9. MYSQL CONNECTORS CONNECTION POOLS Connectors Configuring Connector Creating A Database Connection Prepared Statements Example Transaction 9
- 10. Creating Connection Connection con = DriverManager.getConnection (“jdbc:mysql://node2/employees? user=connj&password=test"); Statement stmt = con.createStatement(); String query = "select * from employees where emp_no = 20000;"; ResultSet rs = stmt.executeQuery(query); ... MariaDB: jdbc:mariadb://node2/employees ?user=connj&password=test" 10
- 11. Creating Connection - Tomcat w. JDBC- Pool context.xml (local): <Resource name="jdbc/test" auth="Container" type="javax.sql.DataSource" username=“jdbc-pool" password="test" driverClassName="com.mysql.jdbc.Driver" url=“jdbc:mysql://node2:3306/employees” /> MariaDB: driverClassName="org.mariadb.jdbc.Driver" 11
- 12. Creating Connection - JDBC URL jdbc:mysql://node2:3306/employees? useServerPrepStmts=true&... 12
- 13. MYSQL CONNECTORS CONNECTION POOLS Connectors Configuring Connector Creating A Database Connection Prepared Statements Example Transaction 13
- 14. Connector/J - Creating Connection SHOW VARIABLES WHERE Variable_name ='language' OR… SELECT @@session.auto_increment_increment; SET NAMES latin1; SET character_set_results = NULL; SET autocommit=1; SET sql_mode= 'NO_ENGINE_SUBSTITUTION,STRICT_TRAN S_TABLES'; 14
- 15. MariaDB - Creating Connection set autocommit=1; USE employees; show variables like 'sql_mode'; 15
- 16. Creating Connection - Defaults • Connector/J: • MariaDB Java Client: 16
- 17. Connector/J & MariaDB Java Client - Verbosity • Connector/J is more verbose when starting a connection • Usually not a problem: – connection pools are commonly used (more coming soon…) – connections are reused – Actually I like but not too much. 17
- 18. Optimization • MariaDB Java Client vs MySQL Connector/J • Prepared Statements 18
- 19. Connector Performance - SELECT 1 localhost, single threaded 19 QPS 0 4000 8000 12000 16000 localhost ConnectorJ MariaDB 15.213 13.477
- 20. Connector Performance - MariaDB %faster 20 Faster% 5% 10% 15% 20% MariaDB Connector +Speed% ConnectorJ SELECT1 LO 13%
- 21. Connector Performance - MariaDB %faster 21 Faster% 5% 10% 15% 20% MariaDB Connector +Speed% ConnectorJ SELECT1 LO SELECT1 net pk range 0% 4%4% 13% Benefit is relative!
- 22. MYSQL CONNECTORS CONNECTION POOLS Connectors Configuring Connector Creating A Database Connection Prepared Statements Example Transaction 22
- 23. Client or Server Side Prepared Statements • Server side Prepared statements: – reduce network traffic – query is already optimized on server. • Support: – MariaDB Java client only supports client side – Connector/J default in client side 23
- 24. Server Side Prepared Statements PREPARE stmt1 FROM select * from employees where emp_no = ?; EXECUTE # API CALL select * from employees where emp_no = 20000; DEALLOCATE PREPARE stmt1; 24
- 25. Connector/J: Server Side Prepared Statements • useServerPrepStmts = false – disabled by default • Mind looking at: – cachePrepStmts = false • do PREPARE, EXECUTE, DEALLOCATE every time…, 3 round trips? – prepStmtCacheSize = 25 – prepStmtCacheSqlLimit = 256 • low defaults 25 https://bugs.mysql.com/bug.php?id=74932
- 26. Benchmark: Prepared Statements 26 QPS 900 1800 2700 3600 MariaDB CONN/J CONN/J SRVCONN/J SRV+Cache 3.506QPS 2.047QPS 3.342QPS3.400QPS select * from employees_alotofindexes where first_name='moss' and birth_date > "1954-06-14" and gender="M" and hire_date > "1998-01-01"G
- 27. Cracked!! 27
- 28. MYSQL CONNECTORS CONNECTION POOLS Connectors Configuring Connector Creating A Database Connection Prepared Statements Example Transaction 28
- 29. Connector/J Optimization + Default JDBC- Pool Connection con = ds.getConnection(); con.setTransactionIsolation (Connection.TRANSACTION_READ_COMMITTED); con.setAutoCommit(false); PreparedStatement stmt = con.prepareStatement("select * from employees where emp_no = ?"); stmt.setInt(1, 20000); ResultSet rs = stmt.executeQuery(); stmt.close(); rs.close(); con.commit(); con.close(); 29
- 30. Connector/J Optimization + Default JDBC- Pool SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED; SET autocommit=0; # administrator command: Prepare; select * from employees where emp_no = 20000; # administrator command: Close stmt; commit; 30 Taxes
- 31. Connector/J Optimization • useConfigs=maxPerformance – cachePrepStmts=true – cacheCallableStmts=true – cacheServerConfiguration=true – useLocalSessionState=true – elideSetAutoCommits=true – alwaysSendSetIsolation=false – enableQueryTimeouts=false 31
- 32. Connector/J Optimization • useLocalTransactionState=true commit() / rollback() 32
- 33. Connector/J Optimization - Tuned JDBC URL: useConfigs=maxPerformance& useServerPrepStmts=true: select * from employees where emp_no = 20000; commit; 33
- 34. MariaDB Java Client Optimization SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED; select * from employees where emp_no = 20000; COMMIT; 34
- 35. MariaDB Java Client Optimization - Code if ( con.getTransactionIsolation() != Connection.TRANSACTION_READ_COMMITTED ) { con.setTransactionIsolation (Connection.TRANSACTION_READ_COMMITTED); } 35
- 36. MariaDB Java Client Optimization - Interceptors SELECT @@tx_isolation; select * from employees where emp_no = 20000; COMMIT; Still @@tx_isolation. Now add JDBC Interceptor: <Resource name="jdbc/test" auth="Container" factory= "org.apache.tomcat.jdbc.pool.DataSourceFactory" jdbcInterceptors="ConnectionState" driverClassName="org.mariadb.jdbc.Driver" url="jdbc:mysql://node2:3306/employees"/> 36
- 37. MariaDB Java Client Optimization - Optimized! select * from employees where emp_no = 20000; COMMIT; 37
- 38. Benchmark - Connector Concurrency - SELECT 1 38 HikariCP-bench with JDBC Pool, 4 Threads, SELECT 1 (4,8,16,32 Pool Size) QPS 17.500 35.000 52.500 70.000 MariaDB CONN/J
- 39. Benchmark - Connector Concurrency - TRX 39 HikariCP-bench with JDBC Pool, 4 Threads, TRX (4,8,16,32 Pool Size) QPS 4.750 9.500 14.250 19.000 Conn/J MariaDB Conn/J Optim MariaDB Optim
- 41. MYSQL CONNECTORS CONNECTION POOLS Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features 41
- 42. Java Connection Pools – The Usual: – C3P0 – Commons-DBCP (v1&v2) – JDBC Pool (fork commons-DBCP) – Out In The Wild: – Vibur-DBCP – HikariCP – BoneCP 42
- 43. Java Connection Pools – The Usual: – C3P0 – Commons-DBCP (v1&v2) – JDBC Pool (fork commons-DBCP) – Out In The Wild: – Vibur-DBCP – HikariCP – BoneCP 43
- 44. Connection Pool Key Points • Connection Management • Pool Sizing • Connection Testing • Avoid Lingering Transactions 44
- 45. MYSQL CONNECTORS CONNECTION POOLS Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features 45
- 46. Connection Pool Issues • Coming from DBA side, I do not like ‘chattiness’ 46
- 51. WHY DOES IT HAVE TO DO THAT? Because of ‘application bugs’ 51
- 52. Connection Pools - Why Chattiness Examples • *maybe* forgot to COMMIT / ROLLBACK • wanting AUTOCOMMIT=1 but a previous TRX set it to 0 • Changing TRX Isolation Level • Is connection still working? 52
- 53. MYSQL CONNECTORS CONNECTION POOLS Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features 53
- 54. Connection Pool - Resetting Status 54 JDBC-Pool C3P0 DBCP2 HikariCP Rollback rollbackOnReturn=false autoCommitOnClose=false rollbackOnReturn =true Commit commitOnReturn=false autoCommitOnClose=false n/a n/a Avoid see above forceIgnoreUnresolvedTransa ctions=false see above Auto Commit Driver Driver enableAutoCommit OnReturn=true Driver
- 55. MYSQL CONNECTORS CONNECTION POOLS Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features 55
- 56. Connection Pool - Testing • Making sure the connection is still active • If not, maybe reopen a connection • Not recommended as DB • However, applications: • do not like errors • do not retry gracefully 56
- 57. Connection Pool - Testing • If connections REALLY need to be tested… • do not specify test query like: • SELECT 1 • SELECT * FROM DUAL • Leave default, all of the connection pools use: JDBC4 isValid(); 57
- 58. Connection Pool - Testing 58 JDBC-Pool C3P0 DBCP2 HikariCP Test Before testOnBorrow=false testConnectionOnCheckOut =false testOnBorrow=false n/a Test After testOnReturn=false testConnectionOnCheckIn =false testOnReturn=false n/a Test While Idle testWhileIdle=false idleConnectionTestPeriod =0 testWhileIdle=false n/a JDBC4 isValid() default default default jdbc4ConnectionTest =true (default) Query validationQuery (isValid) preferredTestQuery=null validationQuery (isValid) connectionTestQuery =none Interval? validationInterval=30000 n/a n/a n/a
- 59. Connection Pool - Testing • JDBC: validationInterval=30s WHY? It defeats the whole purpose! 59
- 60. MYSQL CONNECTORS CONNECTION POOLS Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features 60
- 61. Connection Pool - Pool Sizing • Funnelling on Application Level, is good • Smaller Number is Better • +- * CPU’s on DB • maybe a bit more (waiting on IO…) • all application servers combined • Response Time vs Throughput 61
- 62. Connection Pool - Pool Sizing 62 JDBC-Pool C3P0 DBCP2 HikariCP Amount of Connections maxActive=100 maxPoolSize=15 maxTotal=8 maximumPoolSize=10 Maximum Idle Connections maxIdle=100 maxIdleTime=0** maxIdle=8 n/a Minimum Idle Connections minIdle=10 minPoolSize=3 minIdle=0 minimumIdle=max Startup Size initialSize=10 initialPoolSize=3 initialSize=0 minimumIdle
- 63. MYSQL CONNECTORS CONNECTION POOLS Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features 63
- 64. Connection Pool - Avoid Lingering Transactions • Application forgets to return the connection • Statements that take longer than … • Avoid this! • Fix Application 64
- 65. Connection Pool - Avoid Lingering Transactions 65 KILL Warning JDBC-Pool removeAbandoned=false removeAbandonedTimeout=60 abandonWhenPercentageFull=0 suspectTimeout=0 C3P0 unreturnedConnectionTimeout=0 n/a DBCP removeAbandoned=false removeAbandonedTimeout=300 Only When: getNumIdle() < 2 and getNumActive() > getMaxTotal() - 3) n/a HikariCP n/a leakDetectionThreshold=0
- 66. MYSQL CONNECTORS CONNECTION POOLS Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features 66
- 67. Connection Pools - How To Look At Workload? • Slow Query Log • tcpdump • pt-query-digest • Percona Cloud Tools 67
- 68. MYSQL CONNECTORS CONNECTION POOLS Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features 68
- 69. Connection Pool - Example Transaction Connection con = ds.getConnection(); con.setTransactionIsolation (Connection.TRANSACTION_READ_COMMITTED); con.setAutoCommit(false); PreparedStatement stmt = con.prepareStatement("select * from employees where emp_no = ?"); stmt.setInt(1, 20000); ResultSet rs = stmt.executeQuery(); stmt.close(); rs.close(); con.commit(); con.close(); 69
- 70. For Connectors - RECAP • MySQL Connector/J • useConfigs=maxPerformance • useServerPrepStmts=true • MariaDB Java Client • HikariCP: Built in • JDBC-Pool: jdbcInterceptors=“ConnectionState" • Other Pools: UNKNOWN 70
- 71. Connection Pool - TRX JDBC select * from employees where emp_no = 20000; commit; 71 200
- 72. Connection Pool - TRX C3P0 SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED; SET autocommit=0; select * from employees where emp_no = 20000; commit; SET autocommit=1; SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ; 72 600
- 73. Connection Pool - TRX C3P0 mysql> set global tx_isolation=“READ-COMMITTED”; forceIgnoreUnresolvedTransactions=true 73 200
- 74. Connection Pool - TRX DBCP SET autocommit=1; # administrator command: Ping; SET autocommit=0; select * from employees where emp_no = 20000; commit; rollback; SET autocommit=1; 74 700
- 75. Connection Pool - TRX DBCP testOnBorrow=false rollbackOnReturn=false enableAutoCommitOnReturn=false jdbcUrl: useLocalTransactionState=true 75 200
- 76. Connection Pool - TRX HikariCP SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED; SET autocommit=0; select * from employees where emp_no = 20000; commit; SET autocommit=1; SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ; 76 600
- 77. Connection Pool - TRX HikariCP mysql> set global tx_isolation=“READ-COMMITTED”; autoCommit=false 77 200
- 80. MYSQL CONNECTORS CONNECTION POOLS Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features 80
- 81. Connection Pools - Graceful Failover 81
- 82. Connection Pools - Graceful Failover • HAProxy ‘stats socket’ /etc/haproxy/haproxy.cfg global . . . stats socket /tmp/haproxy.sock level admin • Disable Node # echo "disable server database/node1" | socat stdio /tmp/haproxy.sock 82
- 83. Connection Pools - Graceful Failover 83
- 84. Connection Pools - Graceful Failover • During ‘maintenance’, what do we do? • KILL old connections? • Wait until connections are closed? (Define lifetimes?) • Ignore it? 84
- 85. Connection Pools - Graceful Failover • Some connection pools can close connections gracefully, when idle. • For ‘synchronous’ replication systems • using JMX • No Application Errors! 85 Method JDBC-Pool purgeOnReturn() C3P0 softResetAllUsers() DBCP n/a HikariCP softEvictConnections(), suspendPool(), resumePool() <—- ASYNC
- 86. Connection Pools - Graceful Failover 86
- 87. Connection Pools - Graceful Failover 87
- 88. Connection Pools - Graceful Failover 88
- 89. Connection Pools - Graceful Failover • 0 Application Errors • Completely seamless 89
- 90. MYSQL CONNECTORS CONNECTION POOLS Connection Pools Issues Resetting Environment Testing Connectivity Pool Sizing Lingering Transactions Analysis Examples Graceful Failover Conn/J Extra Features 90
- 91. Connector/J - Extra Features • Load Balancing • jdbcUrl: ”jdbc:mysql:loadbalance:// node1,node2/db? loadBalanceConnectionGroup=lb& loadBalanceEnableJMX=true” • loadBalanceStrategy (random/bestResponseTime) • Failover • ReplicationDriver (setReadOnly) • Combining with Connection Pools is less useful • Fabric 91
- 92. Java MySQL Connector & Connection Pool Optimization • http://dev.mysql.com/doc/connector-j/en • https://mariadb.com/kb/en/mariadb/client-libraries/mariadb-java- client/ • http://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html • http://www.mchange.com/projects/c3p0 • http://commons.apache.org/proper/commons-dbcp/ • https://github.com/brettwooldridge/HikariCP 92 MYSQL CONNECTORS CONNECTION POOLS Kenny Gryp <[email protected]> November 4, 2014 @gryp