![Dump and load using MySQL Shell & Python
This example is inspired by @datacharmer's work: https://www.slideshare.net/datacharmer/mysql-documentstore
$ mongo quiet eval 'DBQuery.shellBatchSize=30000;
db.restaurants.find().shellPrint()'
| perl -pe 's/(?:ObjectId|ISODate)(("[^"]+"))/ $1/g' > all_recs.json
14](https://image.slidesharecdn.com/mysqlwosqldetphp-180728144556/85/MySQL-Without-The-SQL-Oh-My-PHP-Detroit-July-2018-14-320.jpg)
![Find the top 10 restaurants by grade for each cuisine 46
WITH cte1 AS (SELECT doc->>"$.name" AS 'name',
doc->>"$.cuisine" AS 'cuisine',
(SELECT AVG(score) FROM
JSON_TABLE(doc, "$.grades[*]"
COLUMNS (score INT PATH "$.score")) as r ) AS avg_score
FROM restaurants)
SELECT *, rank() OVER
(PARTITION BY cuisine ORDER BY avg_score) AS `rank`
FROM cte1
ORDER by `rank`, avg_score DESC limit 10;
This query uses a Common Table Expression (CTE) and a Windowing Function to rank the average
scores of each restaurant, by each cuisine](https://image.slidesharecdn.com/mysqlwosqldetphp-180728144556/85/MySQL-Without-The-SQL-Oh-My-PHP-Detroit-July-2018-46-320.jpg)
MySQL Without The SQL -- Oh My! PHP Detroit July 2018
- 1. MySQLWithout the SQL - Oh My! Dave Stokes @stoker [email protected] Elephantdolphin.blogger.com opensourcedba.wordpress.com
- 3. Relational Databases ● Data Integrity ○ Normalization ○ constraints (foreign keys, ...) ● Atomicity, Consistency, Isolation, Durability ○ ACID compliant ○ transactions ● SQL ○ powerful query language 3
- 4. NoSQL or Document Store 4
- 5. NoSQL or Document Store ● Schemaless ○ No schema design, no normalization, no foreign keys, no data types, … ○ Very quick initial development ● Flexible data structure ○ Embedded arrays or objects ○ Valid solution when natural data can not be modelized optimally into a relational model ○ Objects persistence without the use of any ORM - *mapping object-oriented* ● JSON ● close to frontend ● native in JS ● easy to learn 5
- 6. How DBAs see data as opposed to how Developers see data { "GNP" : 249704, "Name" : "Belgium", "government" : { "GovernmentForm" : "Constitutional Monarchy, Federation", "HeadOfState" : "Philippe I" }, "_id" : "BEL", "IndepYear" : 1830, "demographics" : { "Population" : 10239000, "LifeExpectancy" : 77.8000030517578 }, "geography" : { "Region" : "Western Europe", "SurfaceArea" : 30518, "Continent" : "Europe" } } 6
- 7. What if there was a way to provide both SQL and NoSQL on one stable platform that has proven stability on well know technology with a large Community and a diverse ecosystem ? With the MySQL Document Store SQL is now optional! 7
- 8. A Solution for all Developers: ★ schemaless ★ rapid prototyping & simpler APIs ★ document model ★ transactions Operations: ★ performance management/visibility ★ robust replication, backup, restore ★ comprehensive tooling ecosystem ★ simpler application schema upgrades 8 Business Owner: ★ don't lose my data == ACID trx ★ capture all my data = extensible/schemaless ★ product on schedule/time to market = rapid development
- 9. Built on the MySQL JSON Data type and Proven MySQL Server Technology 9 ★ Provides a schema flexible JSON Document Store ★ No SQL required ★ No need to define all possible attributes, tables, etc. ★ Uses new X DevAPI ★ Can leverage generated column to extract JSON values into materialized columns that can be indexed for fast SQL searches. ★ Document can be ~1GB ○ It's a column in a row of a table ★ Allows use of modern programming styles ○ No more embedded strings of SQL in your code ○ Easy to read ★ Also works with relational Tables ★ Proven MySQL Technology
- 10. ★ Connectors for ○ C++, Java, .Net, Node.js, Python, PHP ○ working with Communities to help them supporting it too ★ New MySQL Shell ○ Command Completion ○ Python, JavaScripts & SQL modes ○ Admin functions ○ New Util object ○ A new high-level session concept that can scale from single MySQL Server to a multiple server environment ★ Non-blocking, asynchronous calls follow common language patterns ★ Supports CRUD operations 10
- 11. Starting using MySQL in few minutes 11
- 12. Shell info 12
- 13. For this example, I will use the well known restaurants collection: We need to dump the data to a file and we will use the MySQL Shell with the Python interpreter to load the data. Migration from MongoDB to MySQL Document Store 13
- 14. Dump and load using MySQL Shell & Python This example is inspired by @datacharmer's work: https://www.slideshare.net/datacharmer/mysql-documentstore $ mongo quiet eval 'DBQuery.shellBatchSize=30000; db.restaurants.find().shellPrint()' | perl -pe 's/(?:ObjectId|ISODate)(("[^"]+"))/ $1/g' > all_recs.json 14
- 15. 15
- 16. 16 Let’s query Too many records to show here … let’s limit it!
- 18. 18 Let’s add a selection criteria
- 19. > db .r es ta > db.restaurants.find({"cuisine": "French", "borough": { $not: /^Manhattan/} }, {"_id":0, "name": 1,"cuisine": 1, "borough": 1}).limit(2) { "borough" : "Queens", "cuisine" : "French", "name" : "La Baraka Restaurant" } { "borough" : "Queens", "cuisine" : "French", "name" : "Air France Lounge" } 19 Syntax is slightly different than MongoDB
- 25. 25 MySQL Document Store Objects Summary
- 26. MySQL Document Store is Fully ACID Compliant 26
- 27. MySQL Document Store is Fully ACID Compliant 27
- 28. What about old SQL? The Hidden Part of the Iceberg 28
- 29. ★ Native datatype (since 5.7.8) ★ JSON values are stored in MySQL tables using UTF8MB4 ★ Conversion from "native" SQL types to JSON values ★ JSON manipulation functions (JSON_EXTRACT, JSON_KEYS, JSON_SEARCH, JSON_TABLES, ...) ★ Generated/virtual columns ○ Indexing JSON data ○ Foreign Keys to JSON data ○ SQL Views to JSON data JSON datatype is behind the scene 29
- 30. How Does It Work?? 30
- 31. What does a collection look like on the server ? 31
- 32. Every document has a unique identifier called the document ID, which can be thought of as the equivalent of a table´s primary key. The document ID value can be manually assigned when adding a document. If novalue is assigned, a document ID is generated and assigned to the document automatically ! Use getDocumentId() or getDocumentIds() to get _ids(s) _id 32
- 33. Mapping to SQL Examples createCollection('mycollection') CREATE TABLE `test`.`mycoll` ( doc JSON, _id VARCHAR(32) GENERATED ALWAYS AS (doc->>'$._id') STORED PRIMARY KEY ) CHARSET utf8mb4; mycollection.add({‘test’: 1234}) INSERT INTO `test`.`mycoll` (doc) VALUES ( JSON_OBJECT('_id','663807fe367ee6114e0e5458bdac28bf', 'test',1234)); 33
- 34. More Mapping to SQL Examples mycollection.find("test > 100") SELECT doc FROM `test`.`mycoll` WHERE (JSON_EXTRACT(doc,'$.test') >100); 34
- 35. 35 SQL and JSON Example
- 36. It's also possible to create indexes without using SQL syntax 36
- 37. SQL and JSON Example (2): validation 37
- 38. SQL and JSON Example (3): explain 38
- 39. SQL and JSON Example (3): explain 39
- 40. SQL and JSON Example (4): add index 40
- 41. SQL and JSON Example (4): add index 41
- 42. SQL and JSON Example (5): arrays 42
- 43. NoSQL as SQL 43 JSON_TABLE turns your un-structured JSON data into a temporary structured table!
- 44. NoSQL as SQL 44 This temporary structured table can be treated like any other table -- LIMIT, WHERE, GROUP BY ...
- 46. Find the top 10 restaurants by grade for each cuisine 46 WITH cte1 AS (SELECT doc->>"$.name" AS 'name', doc->>"$.cuisine" AS 'cuisine', (SELECT AVG(score) FROM JSON_TABLE(doc, "$.grades[*]" COLUMNS (score INT PATH "$.score")) as r ) AS avg_score FROM restaurants) SELECT *, rank() OVER (PARTITION BY cuisine ORDER BY avg_score) AS `rank` FROM cte1 ORDER by `rank`, avg_score DESC limit 10; This query uses a Common Table Expression (CTE) and a Windowing Function to rank the average scores of each restaurant, by each cuisine
- 47. SQL and NoSQL 47
- 48. Show Results 48
- 50. Conclusion: What Do I Gain? 50
- 51. This is the best of the two worlds in one product ! ● Data integrity ● ACID Compliant ● Transactions ● SQL ● Schemaless ● flexible data structure ● easy to start (CRUD) 51