Test complex database systems in your laptop with dbdeployer
0 likes567 views
Giuseppe Maxia is a software explorer at VMware who has created several tools for MySQL over the decades, including MySQL-Sandbox. He rewrote MySQL-Sandbox in Go to create a new tool called dbdeployer. Dbdeployer allows users to easily install, configure, and manage isolated MySQL sandboxes on their local machine without dependencies. It provides functionality for single server, replication, and group replication deployments through an intuitive command line interface. The tool handles tasks like port allocation and initialization automatically to simplify the sandbox development workflow.
![dbdeployer help
$ dbdeployer --help
dbdeployer makes MySQL server installation an easy task.
Runs single, multiple, and replicated sandboxes.
Usage:
dbdeployer [command]
Available Commands:
admin sandbox management tasks
defaults tasks related to dbdeployer defaults
delete delete an installed sandbox
deploy deploy sandboxes
global Runs a given command in every sandbox
help Help about any command
sandboxes List installed sandboxes
unpack unpack a tarball into the binary directory
usage Shows usage of installed sandboxes
versions List available versions](https://image.slidesharecdn.com/dbdeployer-190202093322/85/Test-complex-database-systems-in-your-laptop-with-dbdeployer-27-320.jpg)
![unpack
$ dbdeployer unpack -h
If you want to create a sandbox from a tarball, you first need to
unpack it into the sandbox-binary directory. This command carries out
that task, so that afterwards you can call 'deploy single', 'deploy
multiple', and 'deploy replication' commands with only the MySQL
version for that tarball.
If the version is not contained in the tarball name, it should be
supplied using --unpack-version.
If there is already an expanded tarball with the same version, a new
one can be differentiated with --prefix.
Usage:
dbdeployer unpack MySQL-tarball [flags]
Flags:
--prefix string Prefix for the final expanded directory
--unpack-version string which version is contained in the
tarball
--verbosity int Level of verbosity during unpack (0-2)](https://image.slidesharecdn.com/dbdeployer-190202093322/85/Test-complex-database-systems-in-your-laptop-with-dbdeployer-28-320.jpg)
![deploy
$ dbdeployer deploy -h
Deploys single, multiple, or replicated
sandboxes
Usage:
dbdeployer deploy [command]
Available Commands:
multiple create multiple sandbox
replication create replication sandbox
single deploys a single sandbox](https://image.slidesharecdn.com/dbdeployer-190202093322/85/Test-complex-database-systems-in-your-laptop-with-dbdeployer-29-320.jpg)
![deploy flags (2)
Flags:
--native-auth-plugin in 8.0.4+, uses the native auth plugin
--port int Overrides default port
--post-grants-sql strings SQL queries after loading grants
--post-grants-sql-file string SQL file after loading grants
--pre-grants-sql strings SQL queries before loading grants
--pre-grants-sql-file string SQL file to run before loading grants
--remote-access string defines the database access ("127.%")
--rpl-password string replication password ("rsandbox")
--rpl-user string replication user ("rsandbox")
--sandbox-directory string Changes the default sandbox directory
--skip-load-grants Does not load the grants
--skip-report-host Does not include report host
--skip-report-port Does not include report port
--skip-start Does not start the database server
--use-template strings [template_name:file_name] Replace
existing template with one from file](https://image.slidesharecdn.com/dbdeployer-190202093322/85/Test-complex-database-systems-in-your-laptop-with-dbdeployer-31-320.jpg)
![deploy single
$ dbdeployer deploy single -h
single installs a sandbox and creates useful scripts for its use.
MySQL-Version is in the format x.x.xx, and it refers to a
directory named after the version
containing an unpacked tarball. The place where these directories
are found is defined by
--sandbox-binary (default: $HOME/opt/mysql.)
For example:
dbdeployer deploy single 5.7.21
For this command to work, there must be a directory $HOME/opt/
mysql/5.7.21, containing
the binary files from mysql-5.7.21-$YOUR_OS-x86_64.tar.gz
Use the "unpack" command to get the tarball into the right
directory.
Usage:
dbdeployer deploy single MySQL-Version [flags]](https://image.slidesharecdn.com/dbdeployer-190202093322/85/Test-complex-database-systems-in-your-laptop-with-dbdeployer-32-320.jpg)
![deploy multiple
$ dbdeployer deploy multiple -h
Creates several sandboxes of the same version, without any
replication relationship.
For this command to work, there must be a directory $HOME/opt/
mysql/5.7.21, containing the binary files from mysql-5.7.21-
$YOUR_OS-x86_64.tar.gz
Use the "unpack" command to get the tarball into the right
directory.
Usage:
dbdeployer deploy multiple MySQL-Version [flags]
Examples:
$ dbdeployer deploy multiple 5.7.21
Flags:
-n, --nodes int How many nodes will be installed (default 3)](https://image.slidesharecdn.com/dbdeployer-190202093322/85/Test-complex-database-systems-in-your-laptop-with-dbdeployer-33-320.jpg)
![deploy replication (1)
$ dbdeployer deploy replication -h
The replication command allows you to deploy several nodes
in replication.
Allowed topologies are "master-slave" for all versions,
and "group", "all-masters", "fan-in"
for 5.7.17+.
For this command to work, there must be a directory $HOME/
opt/mysql/5.7.21, containing
the binary files from mysql-5.7.21-$YOUR_OS-x86_64.tar.gz
Use the "unpack" command to get the tarball into the right
directory.
Usage:
dbdeployer deploy replication MySQL-Version [flags]](https://image.slidesharecdn.com/dbdeployer-190202093322/85/Test-complex-database-systems-in-your-laptop-with-dbdeployer-34-320.jpg)
![Remote sandboxes
(since version 1.16.0)
$ dbdeployer remote list
Files available in https://
raw.githubusercontent.com/datacharmer/mysql-
docker-minimal/master/dbdata/available.json
5.7 -> [5.7.24 5.7.25]
8.0 -> [8.0.12 8.0.13]
4.1 -> [4.1.22]
5.0 -> [5.0.15 5.0.96]
5.1 -> [5.1.72]
5.5 -> [5.5.61 5.5.62]
5.6 -> [5.6.41 5.6.43]
$ dbdeployer remote get 8.0.13
File $HOME/your_path/8.0.13.tar.xz downloaded](https://image.slidesharecdn.com/dbdeployer-190202093322/85/Test-complex-database-systems-in-your-laptop-with-dbdeployer-39-320.jpg)
Test complex database systems in your laptop with dbdeployer
- 1. Giuseppe Maxia Software explorer Test complex database systems in a laptop with dbdeployer
- 2. Who's this guy? About me ‣ Giuseppe Maxia, a.k.a. "The Data Charmer" ‣ Software Explorer at VMware ‣ Several decades development and DB experience ‣ Long timer MySQL community member. ‣ Blog: http://datacharmer.blogspot.com ‣ Twitter: @datacharmer !2
- 3. Disclaimer •None of what I say has anything to do with my company. •I also don’t work for Oracle.
- 4. 2006-2017: MySQL-Sandbox • https://github.com/datacharmer/mysql-sandbox • A cool utility to install many MySQL servers in user space. • MySQL application of the year 2013 • A Perl package. • Somehow difficult to install. • Hard to run when the target system does not have Perl... • Syntax evolved over years (since 2006!), and it's kinda messy. !4
- 5. I had a dream improve MySQL-Sandbox !5
- 6. I had a dream rewrite MySQL-Sandbox in Python! in Ruby! in C++ in Java! just joking in Go! Actually, why not? in Javascript! not even for a second !6
- 8. Why not Perl (6), Python, Ruby, C++? • Lost patience waiting for Perl 6 ... • Tired of wrestling sys admins about Python 2.6/7 vs. 3.x ... • Ruby has magically disappeared ... • I still love C++, but I it has the same problem as the above ones: • Burden of build/installation is on users! Libraries dependencies are hell to solve
- 9. Why Go? • Burden of build is on the developer. • Users need only to download the executable. • No dependencies (except a system able to run MySQL) • Fast development • Incredible wealth of public libraries for everything. • Amazing community. !9
- 10. dbdeployer • Command line tool • No dependencies (single binary per O.S.) • Interface similar to git, docker • Fast! • Runs single database, replication, group replication, multiple deployments, multi-source replication. • Integrated help • Easy to extend. https://github.com/datacharmer/dbdeployer
- 11. What can it do? (1) • Install a single MySQL sandbox • Separated from any existing MySQL server • Completely in user space • Reasonably isolated • Easy to use and administer
- 12. What can it do? (2) • Install a group of sandboxes • Isolated from each other • Without any relationship • With binary logging active
- 13. What can it do? (3) • Install a master-slave replication cluster • 1 master and 2 (or more) slaves • Replication enabled • Optional GTID • Easy tools to run operations on all nodes at once
- 14. What can it do? (4) • Install a group replication cluster (MySQL 5.7 and 8.0) • 3 (or more) nodes • Single-primary or multi-primary • Group replication configured and enabled • Easy tools to run operations on all nodes at once
- 15. What can it do? (5) • Install a multi-source replication cluster (MySQL 5.7-8.0) • 3 (or more) nodes • fan-in or all-masters topologies • Easy tools to run operations on all nodes at once
- 16. What can it do? (6) • Administer sandboxes • List available binaries • List installed sandboxes • start, restart (with options), stop • check status • test • delete
- 17. What can it do? (7) • Customize sandboxes • Customizable initialization and database start • Provide templates for every script • Allow on-the-fly and permanent template replacement • Change most of the default values • Make sandboxes permanent (= can't be deleted)
- 18. What can it do? (8) • Find free ports automatically • Test replication flow • Expose MySQL 8 dictionary tables • Use semi-synch replication • Run SQL command before and after loading grants • enable/disable X-protocol
- 19. Installation 1. Go to https://github.com/datacharmer/dbdeployer 2. Find releases 3. Download the binary for your O.S. 4. Put it in a directory within your $PATH 5. Run it!
- 22. Using it 1. Download a MySQL binary tarball 2. Use dbdeployer to expand it 3. start creating sandboxes
- 25. Principles https://dev.mysql.com/downloads/mysql/get a MySQL binary tarball 1 mysql-5.7.22-yourOS.tar.gz dbdeployer unpack2 $HOME/opt/mysql/5.7.22 dbdeployer deploy single3 $HOME/sandboxes/msb_5_7_22 dbdeployer deploy multiple3 $HOME/sandboxes/multi_msb_5_7_22
- 26. operations: unpack once, use forever $ dbdeployer unpack mysql-8.0.11-linux-glibc2.12-x86_64.tar.gz Unpacking tarball mysql-8.0.11-linux-glibc2.12- x86_64.tar.gz to $HOME/opt/mysql/8.0.11 $ dbdeployer available 8.0.11 $ dbdeployer deploy single 8.0.11 Database installed in $HOME/sandboxes/msb_8_0_11
- 27. dbdeployer help $ dbdeployer --help dbdeployer makes MySQL server installation an easy task. Runs single, multiple, and replicated sandboxes. Usage: dbdeployer [command] Available Commands: admin sandbox management tasks defaults tasks related to dbdeployer defaults delete delete an installed sandbox deploy deploy sandboxes global Runs a given command in every sandbox help Help about any command sandboxes List installed sandboxes unpack unpack a tarball into the binary directory usage Shows usage of installed sandboxes versions List available versions
- 28. unpack $ dbdeployer unpack -h If you want to create a sandbox from a tarball, you first need to unpack it into the sandbox-binary directory. This command carries out that task, so that afterwards you can call 'deploy single', 'deploy multiple', and 'deploy replication' commands with only the MySQL version for that tarball. If the version is not contained in the tarball name, it should be supplied using --unpack-version. If there is already an expanded tarball with the same version, a new one can be differentiated with --prefix. Usage: dbdeployer unpack MySQL-tarball [flags] Flags: --prefix string Prefix for the final expanded directory --unpack-version string which version is contained in the tarball --verbosity int Level of verbosity during unpack (0-2)
- 29. deploy $ dbdeployer deploy -h Deploys single, multiple, or replicated sandboxes Usage: dbdeployer deploy [command] Available Commands: multiple create multiple sandbox replication create replication sandbox single deploys a single sandbox
- 30. deploy flags (1) Flags: (cont.) --base-port int Overrides default base-port (for multiple sandboxes) --binary-version string Specifies the version for basedir directory --bind-address string defines the database bind-address ("127.0.0.1") --concurrent Runs multiple sandbox deployments concurrently --custom-mysqld string Uses an alternative mysqld -p, --db-password string database password (default "msandbox") -u, --db-user string database user (default "msandbox") --defaults strings Change defaults on-the-fly (--defaults=label:value) --disable-mysqlx Disable MySQLX plugin (8.0.11+) --enable-general-log Enables general log for the sandbox (MySQL 5.1+) --enable-mysqlx Enables MySQLX plugin (5.7.12+) --expose-dd-tables In MySQL 8.0+ shows data dictionary tables --force Overwrite destination sandbox --gtid enables GTID --init-general-log uses general log during initialization (MySQL 5.1+) -i, --init-options strings mysqld options to run during initialization --keep-server-uuid Does not change the server UUID --my-cnf-file string Alternative source file for my.sandbox.cnf -c, --my-cnf-options strings mysqld options to add to my.sandbox.cnf
- 31. deploy flags (2) Flags: --native-auth-plugin in 8.0.4+, uses the native auth plugin --port int Overrides default port --post-grants-sql strings SQL queries after loading grants --post-grants-sql-file string SQL file after loading grants --pre-grants-sql strings SQL queries before loading grants --pre-grants-sql-file string SQL file to run before loading grants --remote-access string defines the database access ("127.%") --rpl-password string replication password ("rsandbox") --rpl-user string replication user ("rsandbox") --sandbox-directory string Changes the default sandbox directory --skip-load-grants Does not load the grants --skip-report-host Does not include report host --skip-report-port Does not include report port --skip-start Does not start the database server --use-template strings [template_name:file_name] Replace existing template with one from file
- 32. deploy single $ dbdeployer deploy single -h single installs a sandbox and creates useful scripts for its use. MySQL-Version is in the format x.x.xx, and it refers to a directory named after the version containing an unpacked tarball. The place where these directories are found is defined by --sandbox-binary (default: $HOME/opt/mysql.) For example: dbdeployer deploy single 5.7.21 For this command to work, there must be a directory $HOME/opt/ mysql/5.7.21, containing the binary files from mysql-5.7.21-$YOUR_OS-x86_64.tar.gz Use the "unpack" command to get the tarball into the right directory. Usage: dbdeployer deploy single MySQL-Version [flags]
- 33. deploy multiple $ dbdeployer deploy multiple -h Creates several sandboxes of the same version, without any replication relationship. For this command to work, there must be a directory $HOME/opt/ mysql/5.7.21, containing the binary files from mysql-5.7.21- $YOUR_OS-x86_64.tar.gz Use the "unpack" command to get the tarball into the right directory. Usage: dbdeployer deploy multiple MySQL-Version [flags] Examples: $ dbdeployer deploy multiple 5.7.21 Flags: -n, --nodes int How many nodes will be installed (default 3)
- 34. deploy replication (1) $ dbdeployer deploy replication -h The replication command allows you to deploy several nodes in replication. Allowed topologies are "master-slave" for all versions, and "group", "all-masters", "fan-in" for 5.7.17+. For this command to work, there must be a directory $HOME/ opt/mysql/5.7.21, containing the binary files from mysql-5.7.21-$YOUR_OS-x86_64.tar.gz Use the "unpack" command to get the tarball into the right directory. Usage: dbdeployer deploy replication MySQL-Version [flags]
- 35. deploy replication (2) $ dbdeployer deploy replication -h Examples: $ dbdeployer deploy replication 5.7.21 # (implies topology = master-slave) $ dbdeployer deploy --topology=master-slave replication 5.7.21 # (explicitly setting topology) $ dbdeployer deploy replication --topology=group 5.7.21 $ dbdeployer deploy replication --topology=group 8.0.4 --single-primary $ dbdeployer deploy --topology=all-masters replication 5.7.21 $ dbdeployer deploy --topology=fan-in replication 5.7.21
- 36. deploy replication (3) Flags: --master-ip string Which IP the slaves will connect to ("127.0.0.1") --master-list string Which nodes are masters in a multi-source deployment ("1,2") -n, --nodes int How many nodes (3) --semi-sync Use semi-synchronous --single-primary Using single primary --slave-list string Which nodes are slaves in a multi-source deployment (default "3") -t, --topology string Which topology will be installed
- 37. usage (single) "./start", "./status", "./restart", and "./stop" do what their name suggests. start and restart accept parameters that are eventually passed to the server. e.g.: ./start --server-id=1001 ./restart --event-scheduler=disabled "./use" calls the command line client with the appropriate parameters, Example: ./use -BN -e "select @@server_id" ./use -u root "./clear" stops the server and removes everything from the data directory, letting you ready to start from scratch. (Warning! It's irreversible!) ... and more
- 38. usage (multiple) On a replication sandbox, you have the same commands (run "dbdeployer usage single"), with an "_all" suffix, meaning that you propagate the command to all the members. Then you have "./m" as a shortcut to use the master, "./s1" and "./s2" to access the slaves (and "s3", "s4" ... if you define more). In group sandboxes without a master slave relationship (group replication and multiple sandboxes) the nodes can be accessed by ./n1, ./n2, ./n3, and so on. start_all / status_all / restart_all / stop_all / use_all clear_all / m / s1 / s2 / n1 / n2 The scripts "check_slaves" or "check_nodes" give the status of replication in the sandbox. ... and more
- 39. Remote sandboxes (since version 1.16.0) $ dbdeployer remote list Files available in https:// raw.githubusercontent.com/datacharmer/mysql- docker-minimal/master/dbdata/available.json 5.7 -> [5.7.24 5.7.25] 8.0 -> [8.0.12 8.0.13] 4.1 -> [4.1.22] 5.0 -> [5.0.15 5.0.96] 5.1 -> [5.1.72] 5.5 -> [5.5.61 5.5.62] 5.6 -> [5.6.41 5.6.43] $ dbdeployer remote get 8.0.13 File $HOME/your_path/8.0.13.tar.xz downloaded
- 40. Remote sandboxes • Oracle will publish a repository of minimal tarballs • SOON • (Or so they said one year ago) • Please all: remind MySQL Team at Oracle to do that • Twitter @mysql “Where are the reduced tarballs?” • Cc @lefred
- 41. LIVE DEMO
- 42. ` dbdeployer uses semantic versioning (https://semver.org) Current version: 1.17.1 Version component When does it change? example major Backward Incompatible API changes. (Including new features) 1.21.3 => 2.0.0 minor Backward compatible API changes. Backward compatible new features 1.21.3 => 1.22.0 revision Bug fixes. Code refactoring. No API changes 1.21.3 => 1.21.4
- 43. Next steps • Possible enhancements: • Install sandbox from remote tarball (Just did it!) • Unpack command to slim down tarball contents • Install Docker containers as if they were sandboxes (working on it!)
- 44. Parting thoughts • dbdeployer is open source (Apache license) • Contributions are welcome! https://github.com/datacharmer/dbdeployer