Scaling mysql with python (and Docker).
2 likes1,348 views
Roberto Polli's presentation at EuroPython 2015 discusses scaling MySQL using Python, highlighting utilities, connectors, and replication strategies. He emphasizes the ease of managing MySQL databases with Python and Docker, and presents tools like MySQL Fabric for replication and failover. The talk also covers best practices for provisioning, configuring replication, and utilizing cloud services for MySQL management.
1 of 33
Downloaded 21 times
![MySQL Utilities:
Start with mysqluc for a single entrypoint
• entrypoint for all utilities
• contextual help
• TAB completion
Or access each utility separately.
Specify server URIs as user:pass@host[:port].
Python Utilities Roberto Polli - roberto.polli@par-tec.it](https://image.slidesharecdn.com/main-150705150752-lva1-app6892/85/Scaling-mysql-with-python-and-Docker-8-320.jpg)
![Manage
Setup and administration is done via
mysqlfabric manage [setup|start|ping|stop|teardown]
Configure in /etc/fabric.cfg
• listening ports for xmlrpc service
• admin user (eg. fabric to manage servers)
• applicative user credentials
• db to store fabric data
Setup and start the service
mysqlfabric manage setup
mysqlfabric manage start --daemon
Fabric Roberto Polli - roberto.polli@par-tec.it](https://image.slidesharecdn.com/main-150705150752-lva1-app6892/85/Scaling-mysql-with-python-and-Docker-22-320.jpg)
![Replication Groups
Groups are managed via
mysqlfabric group [create|add|promote|activate] $GROUP
Crete an High Availability group and add a bunch of servers.
mysqlfabric group create $GROUP
mysqlfabric group add $GROUP my-host-1:3306
mysqlfabric group add $GROUP my-host-2:3306
mysqlfabric group add $GROUP my-host-3:3306
Fabric Roberto Polli - roberto.polli@par-tec.it](https://image.slidesharecdn.com/main-150705150752-lva1-app6892/85/Scaling-mysql-with-python-and-Docker-23-320.jpg)
![Replication Groups
Promote one server as a master, eventually picking one.
mysqlfabric group promote $GROUP [--slave_id=UUID]
Show the server group
mysqlfabric group lookup_servers ha
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1
server_uuid address status mode weight
------------ ----------- --------- --------- ------
2cda..110015 172.17.0.21 PRIMARY READ_WRITE 1.0
2cda..110016 172.17.0.22 SECONDARY READ_ONLY 1.0
2cda..110017 172.17.0.23 SECONDARY READ_ONLY 1.0
Fabric Roberto Polli - roberto.polli@par-tec.it](https://image.slidesharecdn.com/main-150705150752-lva1-app6892/85/Scaling-mysql-with-python-and-Docker-24-320.jpg)
Ad
Recommended
Orchestrating MySQL with Python and Docker
Orchestrating MySQL with Python and DockerRoberto Polli The document outlines an agenda for a training session on orchestrating MySQL with Python and Docker. The training will cover introducing Docker and Docker Compose, MySQL replication, MySQL Fabric for managing replication groups, troubleshooting, and provisioning databases. It provides examples of setting up a replicated database environment with Docker Compose and using MySQL Fabric and its Python APIs to configure and manage replication groups, monitor replication, and provision new database instances.
Python for System Administrators
Python for System AdministratorsRoberto Polli This document provides an agenda for a Python training on using Python for system administration tasks. It introduces the presenter and their company Par-Tec. The agenda covers using ipython, path management, encoding, gathering data from the system using modules like psutil and subprocess, parsing files using regular expressions and the /proc filesystem, and processing data using concepts like distributions, deviation and correlation. Exercises are provided for trainees to practice these skills.
Python at Facebook
Python at FacebookAngelo Failla The document outlines how Facebook utilizes Python, emphasizing its central repository for back-end systems and its significance as the third most-used programming language. It details the implementation of Thrift for service definition, the structure of .par files, and the processes for building and running Python-based client-server applications. Additionally, it discusses configuration management, code review practices, and an automation project called 'Keanu' to test system capacity under real-world conditions.
SREConEurope15 - The evolution of the DHCP infrastructure at Facebook
SREConEurope15 - The evolution of the DHCP infrastructure at FacebookAngelo Failla The document describes the evolution of Facebook's DHCP infrastructure. It discusses how Facebook moved from a traditional DHCP architecture with dedicated hardware load balancers to a stateless architecture using the open source DHCP server KEA. With KEA, Facebook is able to distribute DHCP configuration dynamically from an inventory system and extend KEA's functionality through a hook API to integrate DHCP with other internal systems. This improved architecture provides better reliability, scalability, and instrumentation.
FBTFTP: an opensource framework to build dynamic tftp servers
FBTFTP: an opensource framework to build dynamic tftp serversAngelo Failla Angelo Failla, a production engineer at Facebook Ireland, discusses the development of fbtftp, a Python3 framework designed for dynamic TFTP servers. This solution addresses issues related to traditional TFTP protocols, enhancing data center provisioning by eliminating physical load balancers, supporting dynamic configurations, and improving overall efficiency. The framework allows for better statistics tracking and serves files dynamically from local storage, streamlining operations in high-latency environments.
Linux Kernel 개발참여방법과 문화 (Contribution)
Linux Kernel 개발참여방법과 문화 (Contribution)Ubuntu Korea Community This document discusses how open source development works in the Linux kernel project. It begins by explaining that the focus will be on the development mechanisms and open source process rather than technical kernel details. It then provides an overview of how development is done with Git and how the Linux kernel project actively develops through code review via mailing lists and discussion of patch submissions. Examples of the patch submission and review process for the perf subsystem are also given.
Koha installation BALID
Koha installation BALIDNur Ahammad The document outlines a training on the installation and configuration of the Koha Integrated Library System (ILS) from September 3-7, 2013, organized by BALID. It provides a step-by-step guide including installation on Debian, post-installation setup, customization of the OPAC, and essential MySQL commands. Additionally, it covers establishing a cron job setup for proper production use and customizing various pages in Koha's interface.
Lua and its Ecosystem
Lua and its EcosystemFrançois Perrad Lua is a lightweight scripting language embedded in many applications like Wireshark and Redis. It is small but powerful, with features like closures, coroutines, and metatables. Lua is embedded via its C API and allows for extending applications with modules written in Lua. Popular modules include LuaSocket and LuaSQL. Lua sees widespread use due to its small size, speed, portability, and ability to extend large C/C++ applications with scripting.
Software Packaging with RPM
Software Packaging with RPMSchalk Cronjé The document discusses software packaging using RPM (Red Hat Package Manager). It covers RPM's origins, design goals, file format, common commands, and building RPM packages. Key points include that RPM is used by many Linux distributions to manage software installation and updates, and provides features like dependency tracking and package queries.
Configure, Pack and Distribute: An RPM Creation Workshop
Configure, Pack and Distribute: An RPM Creation WorkshopNovell The document outlines a workshop on configuring, packing, and distributing RPMs, covering challenges in software management, an introduction to the RPM build process, and practical exercises for building software. It discusses key aspects of RPMs, including the spec file structure, dependency management, and the installation lifecycle. The final parts focus on different methods of building, including from source, using change-root environments, and working from binary packages.
nftables - the evolution of Linux Firewall
nftables - the evolution of Linux FirewallMarian Marinov This document provides an overview of nftables, the new packet filtering framework that replaces iptables in the Linux kernel. It discusses the history and predecessors to nftables, how nftables works, key differences from iptables like its more flexible table and chain configuration, and examples of basic nftables rulesets. It also covers topics like matches, jumps, load balancing performance, and kernel configuration options for nftables.
zebra & openconfigd Introduction
zebra & openconfigd IntroductionKentaro Ebisawa Zebra is an open source network operation stack that succeeds GNU Zebra and Quagga, designed to work with various protocol and functional modules. It integrates with Openconfigd for data plane management and supports multiple dataplanes, offering CLI capabilities and a Yang model for configuration. The document outlines installation steps, dependencies, and features such as routing, switching, and new protocol support in a modular architecture.
Iptablesrocks
Iptablesrocksqwer_asdf This document provides instructions for maintaining, modifying, and ensuring the automatic startup of an iptables firewall configuration on boot. It discusses monitoring firewall logs for security purposes, and offers examples for opening and closing ports by editing the firewall script and reimporting it. Automatic startup on boot is achieved by editing firewall scripts to run on system initialization.
Lab manual
Lab manualBNilavara This document provides instructions for setting up a workspace to build embedded Linux for the Raspberry Pi 3 board. It describes downloading source code and packages, configuring the build environment, building a basic Linux image, writing it to an SD card, and connecting to the board via serial terminal. Key steps include using Git to obtain source, modifying configuration files, issuing bitbake commands to compile the image, and using minicom to interact with the booted board over USB serial.
Linux Networking Explained
Linux Networking ExplainedThomas Graf This document provides an extensive overview of Linux networking concepts covered in a presentation at LinuxCon 2016. It includes topics such as network devices, namespaces, routing, and various networking technologies like VLANs, bonding, and Open vSwitch. Additionally, it addresses advanced features such as BPF for packet processing and the use of virtual networks, encapsulation methods, and network namespaces in Linux.
Building RT image with Yocto
Building RT image with YoctoAlexandre LAHAYE This document discusses building a real-time (RT) image using Yocto Project. It provides an overview of Yocto and real-time Linux options like PREEMPT_RT and Xenomai. It describes extending Yocto recipes and layers to add support for building RT kernels and images for hardware like Raspberry Pi that can run real-time applications. The document references resources for the Yocto meta-xenomai layer and using Xenomai with Yocto to build real-time enabled embedded Linux images.
Generator Tricks for Systems Programmers
Generator Tricks for Systems ProgrammersDavid Beazley (Dabeaz LLC) The document describes using generators to process large log files. A non-generator solution would open the log file and iterate through it line-by-line, splitting each line to extract the byte value and add it to a running total. However, this requires keeping the entire file contents in memory. A generator-based solution yields lines from the file one at a time to avoid loading the entire file into memory at once.
Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...
Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...Anne Nicolas The document discusses the kernel Application Binary Interface (ABI) and its significance in maintaining compatibility between the kernel and its modules. It highlights challenges with ABI stability, including the impact of changes on out-of-tree modules and the necessity for tooling like libabigail to analyze and track ABI changes. Additionally, it outlines the Android project's approach to ensuring stable ABIs within its kernel builds, emphasizing the role of defining and validating the ABI baseline during development.
Deeper dive in Docker Overlay Networks
Deeper dive in Docker Overlay NetworksLaurent Bernaille The document is a detailed presentation on Docker overlay networks, focusing on the use of VXLAN (Virtual Extensible LAN) as a tunneling technology over UDP to facilitate multi-tenancy in cloud SDN without relying on traditional VLAN limits. It outlines several control plane options for VXLAN, including multicast, point-to-point, user-land VXLAN, and using BGP with EVPN for dynamic address distribution, providing pros and cons for each method. The document includes practical examples, commands, and configurations necessary to set up and manage Docker overlay networks, emphasizing the challenges and solutions in network management.
Kernel Recipes 2019 - GNU poke, an extensible editor for structured binary data
Kernel Recipes 2019 - GNU poke, an extensible editor for structured binary dataAnne Nicolas The document provides an overview of 'GNU Poke', an extensible editor designed for structured binary data released by José E. Marchesi. It discusses the motivation behind its development, the features of the Poke language, its architecture, and future improvements. The document outlines the capabilities of Poke including variable mapping, structs, arrays, and its functioning as a stack machine, while inviting collaboration from the community.
Cisco CCNA OSPF IPV6 Configuration
Cisco CCNA OSPF IPV6 ConfigurationHamed Moghaddam The document provides a step-by-step guide on configuring OSPF for IPv6 using Cisco devices. It includes details on setting up router interfaces, checking neighbor relationships, and verifying configurations through commands like 'show ipv6 route' and 'ping'. Additionally, it emphasizes the importance of enabling IPv6 unicast routing to facilitate proper OSPF operation.
Juniper JNCIA – Juniper Floating Static Route Configuration
Juniper JNCIA – Juniper Floating Static Route ConfigurationHamed Moghaddam The document outlines the configuration of floating static routes on Juniper routers R1 and R2, detailing the necessary IP addresses and commands for establishing connectivity between them. It explains how to set main and backup routes, including adjusting preferences for redundancy and failover scenarios. The configuration allows R1 to ping R2's loopback address while managing inactive routes if the primary connection fails.
Fluentd Hacking Guide at RubyKaigi 2014
Fluentd Hacking Guide at RubyKaigi 2014Naotoshi Seo This document summarizes a talk on hacking the Fluentd log streaming framework. It discusses Fluentd's bootstrap sequence and how it loads plugins. It explains how input plugins pass data to output plugins and how the BufferedOutput plugin works by buffering data to avoid blocking. It cautions that output plugins can block which BufferedOutput avoids, but it can also get stuck if the buffer exceeds capacity. It recommends tuning buffer size and thread counts to improve performance.
Cisco CCNA IPV6 Static Configuration
Cisco CCNA IPV6 Static ConfigurationHamed Moghaddam The document provides a detailed tutorial on configuring IPv6 static routes between two routers, R4 and R10, enabling successful pings to loopback addresses at each router. It outlines step-by-step commands, including enabling unicast routing, configuring loopback interfaces, and adding static routes to establish connectivity. The document concludes with verification commands to ensure the routing configuration works as intended.
2012 coscup - Build your PHP application on Heroku
2012 coscup - Build your PHP application on Herokuronnywang_tw The document discusses deploying PHP applications on Heroku. It provides an overview of Heroku, including that it is a Platform-as-a-Service, was launched in 2007, uses Amazon Web Services, offers many add-ons, allows easy scaling, supports PostgreSQL, and offers some free usage. It then walks through deploying a basic "Hello World" PHP app on Heroku, including creating an app, adding code, committing and pushing to Heroku, and viewing the deployed app.
openPOWERLINK over Xenomai
openPOWERLINK over XenomaiAlexandre LAHAYE This document discusses integrating the openPOWERLINK industrial Ethernet protocol stack with the Xenomai real-time Linux subsystem. It provides background on POWERLINK and Xenomai, describes a proof of concept implementation using Xenomai's RTnet networking stack, and evaluates the performance compared to using the PREEMPT_RT Linux kernel patch. While not yet stable enough for industrial use, the work integrates openPOWERLINK with Xenomai's real-time capabilities and performs better than PREEMPT_RT in initial testing. Future work is needed to further optimize and test the stack.
Sockets and Socket-Buffer
Sockets and Socket-BufferSourav Punoriyar The document provides a detailed explanation of the socket API in Linux for network communication, including how to create sockets using the socket() system call and the various socket types and protocols supported. It describes the data structures involved, such as struct socket and struct sock, as well as the sk_buff structure, which is crucial for handling network packets. Additionally, the document outlines the basic functions required for network I/O and explains the client-server data flow in socket communication.
The Internals of "Hello World" Program
The Internals of "Hello World" ProgramNational Cheng Kung University The document discusses how a "Hello World" program works behind the scenes. It covers topics like compilation, linking, executable file formats, loading programs into memory, and process creation. The key points are:
1) A C program is compiled into an object file, then linked with library files to create an executable. The linker resolves symbols and relocates addresses.
2) Executable files use formats like ELF that contain machine code, data, symbol tables, and sections. Object files have a similar format.
3) When a program runs, the OS loads pages of the executable into memory as needed and sets up the process with its own virtual address space.
4) System calls
Www Kitebird Com Articles Pydbapi Html Toc 1
Www Kitebird Com Articles Pydbapi Html Toc 1AkramWaseem The document discusses using Python and the DB-API module to write scripts that interact with MySQL databases. It provides an example of a short script that connects to a MySQL database, issues a query to get the server version, and prints the result. It then discusses a more extensive script that handles errors, issues different statement types like INSERT and SELECT, and retrieves result rows using fetchone() and fetchall() methods. The document provides information on installing prerequisites like Python and MySQLdb, and executing the example scripts.
Rethink db with Python
Rethink db with PythonPrabhu Raghav This document summarizes a presentation about RethinkDB, an open-source database built on JSON documents. It discusses key features of RethinkDB like supporting queries, being distributed, and ease of administration. It also compares RethinkDB to other NoSQL databases. The document provides examples of using the ReQL query language in Python to create and query a database. It demonstrates how to perform operations like counts, filters, and map-reduce queries.
More Related Content
What's hot (20)
Software Packaging with RPM
Software Packaging with RPMSchalk Cronjé The document discusses software packaging using RPM (Red Hat Package Manager). It covers RPM's origins, design goals, file format, common commands, and building RPM packages. Key points include that RPM is used by many Linux distributions to manage software installation and updates, and provides features like dependency tracking and package queries.
Configure, Pack and Distribute: An RPM Creation Workshop
Configure, Pack and Distribute: An RPM Creation WorkshopNovell The document outlines a workshop on configuring, packing, and distributing RPMs, covering challenges in software management, an introduction to the RPM build process, and practical exercises for building software. It discusses key aspects of RPMs, including the spec file structure, dependency management, and the installation lifecycle. The final parts focus on different methods of building, including from source, using change-root environments, and working from binary packages.
nftables - the evolution of Linux Firewall
nftables - the evolution of Linux FirewallMarian Marinov This document provides an overview of nftables, the new packet filtering framework that replaces iptables in the Linux kernel. It discusses the history and predecessors to nftables, how nftables works, key differences from iptables like its more flexible table and chain configuration, and examples of basic nftables rulesets. It also covers topics like matches, jumps, load balancing performance, and kernel configuration options for nftables.
zebra & openconfigd Introduction
zebra & openconfigd IntroductionKentaro Ebisawa Zebra is an open source network operation stack that succeeds GNU Zebra and Quagga, designed to work with various protocol and functional modules. It integrates with Openconfigd for data plane management and supports multiple dataplanes, offering CLI capabilities and a Yang model for configuration. The document outlines installation steps, dependencies, and features such as routing, switching, and new protocol support in a modular architecture.
Iptablesrocks
Iptablesrocksqwer_asdf This document provides instructions for maintaining, modifying, and ensuring the automatic startup of an iptables firewall configuration on boot. It discusses monitoring firewall logs for security purposes, and offers examples for opening and closing ports by editing the firewall script and reimporting it. Automatic startup on boot is achieved by editing firewall scripts to run on system initialization.
Lab manual
Lab manualBNilavara This document provides instructions for setting up a workspace to build embedded Linux for the Raspberry Pi 3 board. It describes downloading source code and packages, configuring the build environment, building a basic Linux image, writing it to an SD card, and connecting to the board via serial terminal. Key steps include using Git to obtain source, modifying configuration files, issuing bitbake commands to compile the image, and using minicom to interact with the booted board over USB serial.
Linux Networking Explained
Linux Networking ExplainedThomas Graf This document provides an extensive overview of Linux networking concepts covered in a presentation at LinuxCon 2016. It includes topics such as network devices, namespaces, routing, and various networking technologies like VLANs, bonding, and Open vSwitch. Additionally, it addresses advanced features such as BPF for packet processing and the use of virtual networks, encapsulation methods, and network namespaces in Linux.
Building RT image with Yocto
Building RT image with YoctoAlexandre LAHAYE This document discusses building a real-time (RT) image using Yocto Project. It provides an overview of Yocto and real-time Linux options like PREEMPT_RT and Xenomai. It describes extending Yocto recipes and layers to add support for building RT kernels and images for hardware like Raspberry Pi that can run real-time applications. The document references resources for the Yocto meta-xenomai layer and using Xenomai with Yocto to build real-time enabled embedded Linux images.
Generator Tricks for Systems Programmers
Generator Tricks for Systems ProgrammersDavid Beazley (Dabeaz LLC) The document describes using generators to process large log files. A non-generator solution would open the log file and iterate through it line-by-line, splitting each line to extract the byte value and add it to a running total. However, this requires keeping the entire file contents in memory. A generator-based solution yields lines from the file one at a time to avoid loading the entire file into memory at once.
Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...
Kernel Recipes 2019 - Analyzing changes to the binary interface exposed by th...Anne Nicolas The document discusses the kernel Application Binary Interface (ABI) and its significance in maintaining compatibility between the kernel and its modules. It highlights challenges with ABI stability, including the impact of changes on out-of-tree modules and the necessity for tooling like libabigail to analyze and track ABI changes. Additionally, it outlines the Android project's approach to ensuring stable ABIs within its kernel builds, emphasizing the role of defining and validating the ABI baseline during development.
Deeper dive in Docker Overlay Networks
Deeper dive in Docker Overlay NetworksLaurent Bernaille The document is a detailed presentation on Docker overlay networks, focusing on the use of VXLAN (Virtual Extensible LAN) as a tunneling technology over UDP to facilitate multi-tenancy in cloud SDN without relying on traditional VLAN limits. It outlines several control plane options for VXLAN, including multicast, point-to-point, user-land VXLAN, and using BGP with EVPN for dynamic address distribution, providing pros and cons for each method. The document includes practical examples, commands, and configurations necessary to set up and manage Docker overlay networks, emphasizing the challenges and solutions in network management.
Kernel Recipes 2019 - GNU poke, an extensible editor for structured binary data
Kernel Recipes 2019 - GNU poke, an extensible editor for structured binary dataAnne Nicolas The document provides an overview of 'GNU Poke', an extensible editor designed for structured binary data released by José E. Marchesi. It discusses the motivation behind its development, the features of the Poke language, its architecture, and future improvements. The document outlines the capabilities of Poke including variable mapping, structs, arrays, and its functioning as a stack machine, while inviting collaboration from the community.
Cisco CCNA OSPF IPV6 Configuration
Cisco CCNA OSPF IPV6 ConfigurationHamed Moghaddam The document provides a step-by-step guide on configuring OSPF for IPv6 using Cisco devices. It includes details on setting up router interfaces, checking neighbor relationships, and verifying configurations through commands like 'show ipv6 route' and 'ping'. Additionally, it emphasizes the importance of enabling IPv6 unicast routing to facilitate proper OSPF operation.
Juniper JNCIA – Juniper Floating Static Route Configuration
Juniper JNCIA – Juniper Floating Static Route ConfigurationHamed Moghaddam The document outlines the configuration of floating static routes on Juniper routers R1 and R2, detailing the necessary IP addresses and commands for establishing connectivity between them. It explains how to set main and backup routes, including adjusting preferences for redundancy and failover scenarios. The configuration allows R1 to ping R2's loopback address while managing inactive routes if the primary connection fails.
Fluentd Hacking Guide at RubyKaigi 2014
Fluentd Hacking Guide at RubyKaigi 2014Naotoshi Seo This document summarizes a talk on hacking the Fluentd log streaming framework. It discusses Fluentd's bootstrap sequence and how it loads plugins. It explains how input plugins pass data to output plugins and how the BufferedOutput plugin works by buffering data to avoid blocking. It cautions that output plugins can block which BufferedOutput avoids, but it can also get stuck if the buffer exceeds capacity. It recommends tuning buffer size and thread counts to improve performance.
Cisco CCNA IPV6 Static Configuration
Cisco CCNA IPV6 Static ConfigurationHamed Moghaddam The document provides a detailed tutorial on configuring IPv6 static routes between two routers, R4 and R10, enabling successful pings to loopback addresses at each router. It outlines step-by-step commands, including enabling unicast routing, configuring loopback interfaces, and adding static routes to establish connectivity. The document concludes with verification commands to ensure the routing configuration works as intended.
2012 coscup - Build your PHP application on Heroku
2012 coscup - Build your PHP application on Herokuronnywang_tw The document discusses deploying PHP applications on Heroku. It provides an overview of Heroku, including that it is a Platform-as-a-Service, was launched in 2007, uses Amazon Web Services, offers many add-ons, allows easy scaling, supports PostgreSQL, and offers some free usage. It then walks through deploying a basic "Hello World" PHP app on Heroku, including creating an app, adding code, committing and pushing to Heroku, and viewing the deployed app.
openPOWERLINK over Xenomai
openPOWERLINK over XenomaiAlexandre LAHAYE This document discusses integrating the openPOWERLINK industrial Ethernet protocol stack with the Xenomai real-time Linux subsystem. It provides background on POWERLINK and Xenomai, describes a proof of concept implementation using Xenomai's RTnet networking stack, and evaluates the performance compared to using the PREEMPT_RT Linux kernel patch. While not yet stable enough for industrial use, the work integrates openPOWERLINK with Xenomai's real-time capabilities and performs better than PREEMPT_RT in initial testing. Future work is needed to further optimize and test the stack.
Sockets and Socket-Buffer
Sockets and Socket-BufferSourav Punoriyar The document provides a detailed explanation of the socket API in Linux for network communication, including how to create sockets using the socket() system call and the various socket types and protocols supported. It describes the data structures involved, such as struct socket and struct sock, as well as the sk_buff structure, which is crucial for handling network packets. Additionally, the document outlines the basic functions required for network I/O and explains the client-server data flow in socket communication.
The Internals of "Hello World" Program
The Internals of "Hello World" ProgramNational Cheng Kung University The document discusses how a "Hello World" program works behind the scenes. It covers topics like compilation, linking, executable file formats, loading programs into memory, and process creation. The key points are:
1) A C program is compiled into an object file, then linked with library files to create an executable. The linker resolves symbols and relocates addresses.
2) Executable files use formats like ELF that contain machine code, data, symbol tables, and sections. Object files have a similar format.
3) When a program runs, the OS loads pages of the executable into memory as needed and sets up the process with its own virtual address space.
4) System calls
Viewers also liked (20)
Www Kitebird Com Articles Pydbapi Html Toc 1
Www Kitebird Com Articles Pydbapi Html Toc 1AkramWaseem The document discusses using Python and the DB-API module to write scripts that interact with MySQL databases. It provides an example of a short script that connects to a MySQL database, issues a query to get the server version, and prints the result. It then discusses a more extensive script that handles errors, issues different statement types like INSERT and SELECT, and retrieves result rows using fetchone() and fetchall() methods. The document provides information on installing prerequisites like Python and MySQLdb, and executing the example scripts.
Rethink db with Python
Rethink db with PythonPrabhu Raghav This document summarizes a presentation about RethinkDB, an open-source database built on JSON documents. It discusses key features of RethinkDB like supporting queries, being distributed, and ease of administration. It also compares RethinkDB to other NoSQL databases. The document provides examples of using the ReQL query language in Python to create and query a database. It demonstrates how to perform operations like counts, filters, and map-reduce queries.
Succumbing to the Python in Financial Markets
Succumbing to the Python in Financial Marketsdcerezo This document discusses the advantages of using Python for financial markets and algorithmic trading. It provides code samples for implementing combo orders, basket options, and Bermuda swaps in Python using various libraries. It also discusses some challenges, such as dynamic typing leading to runtime errors, and the importance of forward testing strategies rather than backtesting to avoid overfitting.
"PostgreSQL and Python" Lightning Talk @EuroPython2014
"PostgreSQL and Python" Lightning Talk @EuroPython2014Henning Jacobs PL/Python allows users to write PostgreSQL functions and procedures using Python. It enables accessing PostgreSQL data and running Python code from within SQL queries. For example, a function could query a database table, process the results in Python by accessing modules, and return a value to the SQL query. This opens up possibilities to leverage Python's extensive libraries and expressiveness to expose data and perform complex validation from PostgreSQL.
MySQL User Conference 2009: Python and MySQL
MySQL User Conference 2009: Python and MySQLTed Leung This document discusses various Python database technologies including:
1. Python-DB and the DB-API standard for database access from Python. It also discusses MySQLdb for MySQL access.
2. Using MySQLdb to connect to a MySQL database and execute queries to retrieve and fetch data.
3. The Django web framework's ORM and how it implements the Active Record pattern for mapping database tables to model classes.
4. Other ORMs like SQLObject which also uses the Active Record pattern, and SQLAlchemy which provides both low-level SQL and a high-level ORM.
Fabric (python)
Fabric (python)Fanani M. Ihsan Fabric is a Python library and command line tool that allows developers to automate and streamline SSH deployment and system administration tasks. It provides features like rsync, environment variable management, directory support, and integration with Django to help developers write deployment scripts once and reuse them across multiple servers. The document demonstrates how to install Fabric using pip, create a fabfile.py script, and provides examples of Fabric features and a demo project for deploying Django applications.
Fabric-让部署变得简单
Fabric-让部署变得简单Eric Lo Fabric is a Python library and command-line tool for automating system administration tasks over SSH. It allows users to define Python functions for common operations and execute them remotely on multiple servers. Fabric has APIs for executing commands via SSH, uploading/downloading files, and abstracting servers into roles for easy management. It provides a Pythonic interface for deployment and administration of remote systems.
fabfile.py
fabfile.pyCorey Oordt The document defines functions for managing websites and servers using Fabric. It includes functions for provisioning environments, checking server requirements, installing packages, managing sites through tasks like cleaning, getting info, listing contents, pushing changes, enabling/disabling sites in Apache, and restarting/reloading Apache. Role definitions separate servers into groups like web, media, staging, testing, and database.
Automation - fabric, django and more
Automation - fabric, django and moreIlian Iliev This document discusses how automation with tools like Fabric and Python can make developers, QAs, and admins more efficient. It introduces Fabric, a Python library for automating system administration tasks and deployment over SSH. Fabric allows tasks like server setup, project creation, and deployment to be automated. An example shows using Fabric to add a new user. The document demonstrates how Fabric scripts can be used to automate starting a new Django project on a server, setting it up, deploying the code, and running the application. Links are provided to Fabric documentation and example scripts.
Fabric - a server management tool from Instagram
Fabric - a server management tool from InstagramJay Ren Fabric is a Python tool for streamlining SSH administration tasks. It allows executing commands on remote servers, uploading/downloading files, and defining reusable tasks. Instagram uses it to deploy code and manage infrastructure across hundreds of EC2 instances. The demo showed how Fabric can define tasks, execute commands concurrently on multiple servers, and provide output levels and context managers for task control.
DevOps with Fabric
DevOps with FabricSimone Federici This document provides examples of using Fabric, a Python library for streamlining the use of SSH for application deployment or systems/network administration tasks. Some key points:
- Fabric allows defining tasks that can be run remotely on SSH-accessible servers via simple Python functions decorated with @task.
- It supports running tasks selectively on hosts or roles. Dynamic host lists can be used by querying external data stores.
- Tasks can be composed together and executed sequentially or in parallel.
- Fabric integrates with tools like Django, handles SSH authentication, and provides utilities like file transfers, command line output formatting, and more.
- The examples demonstrate common patterns for application deployment, database migrations, static file management
Python for Derivative Analytics
Python for Derivative AnalyticsAlicia G This document discusses using the Python library DX for modeling complex derivatives and portfolios. It describes simulation classes for modeling relevant risk factors like geometric Brownian motion, jump diffusion, and square root diffusion. It also mentions calibration and hedging strategies like dynamic delta hedging to reduce risk from changes in the underlying asset's price.
Relational Database Access with Python ‘sans’ ORM
Relational Database Access with Python ‘sans’ ORM Mark Rees This document discusses various approaches for accessing relational databases from Python, including ORM libraries like Django and SQLAlchemy, raw SQL queries using the Python DB-API standard, and template libraries like SpringPython. It provides code examples for common database operations like SELECT, INSERT, and comparing database schemas. A number of Python database adapters are also described, supporting databases like PostgreSQL, MySQL, SQLite and more.
Python Deployment with Fabric
Python Deployment with Fabricandymccurdy The document discusses using Fabric for deployment and system administration tasks across multiple servers. It provides examples of Fabric configuration, defining roles for servers, writing tasks to run commands on servers, and how to structure tasks for a full deployment workflow. Fabric allows running commands remotely via SSH and provides tools for task composition and failure handling.
Pythonic Deployment with Fabric 0.9
Pythonic Deployment with Fabric 0.9Corey Oordt The document discusses the deployment automation tool Fabric version 0.9, emphasizing its use for managing different environments such as testing, staging, and production across multiple computers. It outlines the installation process, task definition, and various Fabric commands, including examples of how to clean and manage files on remote hosts. Additionally, it presents usage scenarios where Fabric can streamline server management and deployment tasks.
Programming with Python and PostgreSQL
Programming with Python and PostgreSQLPeter Eisentraut This document discusses using Python to connect to and interact with a PostgreSQL database. It covers:
- Popular Python database drivers for PostgreSQL, including Psycopg which is the most full-featured.
- The basics of connecting to a database, executing queries, and fetching results using the DB-API standard. This includes passing parameters, handling different data types, and error handling.
- Additional Psycopg features like server-side cursors, transaction handling, and custom connection factories to access columns by name rather than number.
In summary, it provides an overview of using Python with PostgreSQL for both basic and advanced database operations from the Python side.
Python Utilities for Managing MySQL Databases
Python Utilities for Managing MySQL DatabasesMats Kindahl The document outlines the MySQL utilities, a collection of Python tools for managing MySQL databases, part of MySQL Workbench 5.2.31 and available under the GPLv2 license. It discusses various utilities for tasks such as database comparison, data export/import, replication setup, and server information retrieval, providing a structured overview of their functionality and usage. Additionally, it includes examples of command executions and guides for accessing and utilizing these utilities effectively.
PostgreSQLとPythonとSQL
PostgreSQLとPythonとSQLSatoshi Yamada SQLやDBの簡単な概要とPythonの親和性、FDW等について
の資料です。
みんなのPython勉強会#19 で使用しました。
https://startpython.connpass.com/event/37958/
Fabric
FabricCalvin Cheng Fabric is a Python library and command-line tool for streamlining system administration tasks like configuration management, deployment, and more. It executes commands locally or remotely over SSH. Fabric allows defining Python functions for tasks in a fabfile.py and running them with the fab command. It helps automate server setup, deployment, maintenance and other operations across multiple servers defined by roles.
Ad
Similar to Scaling mysql with python (and Docker). (20)
DBA だってもっと効率化したい！〜最近の自動化事情とOracle Database〜
DBA だってもっと効率化したい！〜最近の自動化事情とOracle Database〜Michitoshi Yoshida This document provides an overview of infrastructure automation tools like Chef, Puppet, Ansible, Serverspec, Infrataster and the Infrataster OracleDB plugin. It includes steps to setup the Oracle Instant Client, install necessary Ruby gems, write an Infrataster spec test to query the Oracle database and assert that the db_block_size parameter equals 8192, and execute the spec test with RSpec.
MySQL InnoDB Cluster / ReplicaSet - Tutorial
MySQL InnoDB Cluster / ReplicaSet - TutorialMiguel Araújo The document outlines the evolution of MySQL replication technologies, from manual setups to advanced features in MySQL InnoDB Cluster and InnoDB ReplicaSet, emphasizing automation, ease of use, and integration of components. It describes the transition to automated membership management, consistency, and routing within MySQL servers, highlighting tools like MySQL Shell and Router for smoother operations. Additionally, it presents a tutorial on configuring and managing a replica set, including failover procedures and topology management.
MySQL InnoDB Cluster / ReplicaSet - Tutorial
MySQL InnoDB Cluster / ReplicaSet - TutorialKenny Gryp The document outlines the evolution of MySQL replication, highlighting the transition from manual topology setup to automated features like MySQL InnoDB Cluster and InnoDB ReplicaSet which simplify user experience and provide high availability. It details the components involved, integration capabilities, and advantages of using MySQL shell for managing these systems. Additionally, it includes hands-on tutorials for configuring and interacting with MySQL InnoDB ReplicaSet, emphasizing ease of use and efficiency for database administrators.
Percona toolkit
Percona toolkitKarwin Software Solutions LLC The Percona Toolkit is an open-source collection of scripts designed to assist DBAs and developers in various tasks such as profiling, monitoring, and schema changes for MySQL databases. It includes over thirty tools, such as pt-duplicate-key-checker and pt-online-schema-change, each addressing specific database management challenges. The toolkit is intended to improve productivity and reduce pitfalls associated with database programming.
MySQL 5.7 innodb_enhance_partii_20160527
MySQL 5.7 innodb_enhance_partii_20160527Saewoong Lee The document discusses enhancements in MySQL 5.7 related to InnoDB, including features like online DDL, tablespace management, and support for bulk load operations. It also covers buffer pool resizing and settings for undo log management, as well as improvements for handling general tablespaces. Additionally, it explains the online alter table support and outlines various commands and queries used to manage these enhancements.
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo The document provides an overview of the MySQL webinar presented by Miguel Araújo, focusing on MySQL database architectures, including MySQL Shell and Router functionalities. It outlines recovery objectives, deployment strategies, advanced features, and the latest additions to MySQL versions 8.1.0 and 8.2.0. Additionally, it discusses various architectures such as InnoDB clusters, replication strategies, and connection thread management for improved performance.
User Camp High Availability Presentation
User Camp High Availability Presentationsankalita chakraborty This document discusses using Orchestrator and ProxySQL to achieve high availability (HA) on MySQL. It describes how Orchestrator can detect failures, promote a slave to master, and notify ProxySQL of the change. ProxySQL acts as a load balancer and can route reads and writes accordingly. The solution allows automated failover with less than 10 seconds of downtime and no manual intervention. It also supports topologies like replication clusters and can manage multiple MySQL servers from a single Orchestrator instance.
The SaltStack Pub Crawl - Fosscomm 2016
The SaltStack Pub Crawl - Fosscomm 2016effie mouzeli SaltStack is an open source configuration management and orchestration tool. It allows users to provision, deploy, and manage infrastructure and applications across multiple cloud and virtual systems. SaltStack uses a master-minion architecture with a master server to control and manage minions (agent servers). Key features include remote execution, configuration management, orchestration abilities, and a communication bus to manage infrastructure at scale.
Build Automation 101
Build Automation 101Martin Jackson The document discusses various tools and frameworks for efficient system deployment and management, including Cobbler for network installation of Linux, Puppet for configuration management, and Vagrant for creating virtual environments. It also covers tools for orchestration, such as Marionette Collective, and version control using Subversion, along with project management using Redmine and CI tools like Jenkins. Additionally, it highlights best practices for deployment speed, externalizing configuration, and managing software dependencies.
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUA
10 Lessons Learned from using Kafka in 1000 microservices - ScalaUANatan Silnitsky The document outlines key lessons learned from using Kafka at Wix for over 1000 microservices, emphasizing the need for common infrastructure, self-service tools, and handling increased traffic and metadata. It discusses solutions for service management and error handling while advocating for frameworks like Greyhound that streamline Kafka interactions. Best practices for Kafka usage, including monitoring, testing, and event-driven architectures, are also highlighted to support a scalable system.
MySQL Replication Basics -Ohio Linux Fest 2016
MySQL Replication Basics -Ohio Linux Fest 2016Dave Stokes The document discusses the basics of MySQL replication, covering how data is copied from one server to another, the different types of replication (statement-based, row-based, and mixed-format), and the enhancements in MySQL versions up to 5.7. It emphasizes the importance of understanding replication topologies and configuration, including synchronous and asynchronous methods, global transaction identifiers (GTIDs), and setting up replication. Additionally, it provides practical commands for configuring replication and troubleshooting tools for ensuring data consistency across servers.
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné The document discusses MySQL parallel replication, emphasizing the transition in terminology, key features introduced in versions 5.7 and 8.0, and how logical clock mechanisms improve replication efficiency. It explains the functionality of multi-threaded replication, its challenges, and solutions for maintaining data consistency during parallel transactions. Additionally, it provides insights into how transactions are scheduled and executed on replicas using dependency tracking information from binary logs, along with performance optimization techniques such as commit timestamps and group commit.
Curso de MySQL 5.7
Curso de MySQL 5.7Eduardo Legatti The document provides a comprehensive overview of MySQL, including its history, installation steps, and configuration processes. It details MySQL's acquisition by Sun Microsystems and later by Oracle, as well as essential commands for managing databases and user privileges. Additionally, the document discusses various technical elements such as transaction isolation levels, backup methods, and performance tuning.
Introduction to MariaDB
Introduction to MariaDBJongJin Lee The document provides a comprehensive guide on MariaDB covering its features, installation process, and differences from MySQL. It includes sections on licensing, processes, memory, data types, storage engines, and security measures. Additionally, it outlines detailed installation steps and configurations needed to set up MariaDB properly.
Chicago Docker Meetup Presentation - Mediafly
Chicago Docker Meetup Presentation - MediaflyMediafly This document discusses how Bryan Murphy uses Docker at his company Mediafly. It begins by introducing Bryan and his background. It then describes what Mediafly does, including content management systems, secure content delivery, document and video processing, and customizable user interfaces. The document highlights aspects of Mediafly that make it interesting, such as being multi-device, multi-tenant, service oriented, and distributed. It provides examples of technologies used at Mediafly and some key metrics. The document then discusses why Docker is used at Mediafly, covering benefits like being developer friendly, enabling faster iteration and testing, managing dependencies, sharing environments, standardization, isolation, and infrastructure freedom.
MySQL Utilities -- Cool Tools For You: PHP World Nov 16 2016
MySQL Utilities -- Cool Tools For You: PHP World Nov 16 2016Dave Stokes The document discusses MySQL utilities designed for server maintenance and administration, encapsulating complex commands into simpler operations. It highlights the features of MySQL 8 and various utilities, such as database comparison, failover management, and user cloning, aimed at automating tasks and improving efficiency. The utilities are extendable, available under GPLv2, and separate from the MySQL server, included in MySQL Workbench.
MySQL Replication Update -- Zendcon 2016
MySQL Replication Update -- Zendcon 2016Dave Stokes The document provides a comprehensive overview of MySQL replication, discussing its history, architecture, features, and the latest developments as of MySQL 8. Key topics include binary logging, statement and row-based replication, multi-threading, synchronization methods, and best practices for setting up replication with global transaction identifiers (GTIDs). Additionally, various replication topologies and tools for monitoring and testing replication setups are highlighted.
MongoDB.local Austin 2018: MongoDB Ops Manager + Kubernetes
MongoDB.local Austin 2018: MongoDB Ops Manager + KubernetesMongoDB The document outlines the deployment of MongoDB Enterprise database services on Kubernetes, detailing technologies, tools, and best practices for building data-driven applications. It includes a demo for installing the MongoDB operator, running a replica set, and advanced configurations, with steps for connecting to the MongoDB deployment. Additionally, it covers release plans and resources for further involvement and learning in the MongoDB Kubernetes ecosystem.
The Switch as a Server - PuppetConf 2014
The Switch as a Server - PuppetConf 2014Puppet The document discusses Cumulus Linux, an open-source network operating system that allows network switches to be managed like Linux servers. It can be installed on switches using ONIE (Open Network Install Environment) and configured via Zero Touch Provisioning (ZTP) using scripts. Ifupdown2 is introduced as the new interface configuration manager for Cumulus Linux, which handles dependencies and allows templates to configure many interfaces at once.
Oracle 11g R2 RAC setup on rhel 5.0
Oracle 11g R2 RAC setup on rhel 5.0Santosh Kangane This document provides instructions for implementing an Oracle 11g R2 Real Application Cluster on a Red Hat Enterprise Linux 5.0 system using a two-node configuration. It describes pre-installation steps including hardware and network configuration, installing prerequisite packages and libraries, and configuring the Oracle ASM library driver. Detailed steps are provided for installing Oracle Grid Infrastructure and database software, and configuring the single client access name and storage area network.
Ad
More from Roberto Polli (20)
Ratelimit Headers for HTTP
Ratelimit Headers for HTTPRoberto Polli The document discusses ratelimit headers that communicate service limits to clients, aiming to prevent throttling and reduce the proliferation of varied header fields. It proposes standardized headers to convey multiple ratelimit policies and emphasizes the need for a unified approach to enhance client compatibility across different API gateways. Open issues for further input and technical considerations are also highlighted.
Interoperability rules for an European API ecosystem: do we still need SOAP?
Interoperability rules for an European API ecosystem: do we still need SOAP?Roberto Polli The document discusses moving public services in Italy to standardized HTTP APIs using REST instead of SOAP. It outlines the issues with the old SOAP framework, such as high costs and complexity. The new framework uses HTTP semantics like status codes and avoids processing message bodies. Standards are proposed for areas like dates/times, identifiers, errors and service reliability. Future steps include common metrics, digital signatures, and header standardization.
Docker - virtualizzazione leggera
Docker - virtualizzazione leggeraRoberto Polli Il documento tratta la virtualizzazione leggera, con un focus sui container Docker e sui concetti fondamentali riguardanti il kernel e l'isolamento dei processi. Viene spiegato come il kernel gestisce l'accesso alle risorse del computer e come la virtualizzazione permette a più utenti di utilizzare lo stesso hardware in modo economico. Si approfondisce l'uso dei container come strumenti per isolare ambienti di esecuzione senza necessità di kernel separati.
Just one-shade-of-openstack
Just one-shade-of-openstackRoberto Polli This document discusses using the Shade library to simplify working with OpenStack. It begins with an introduction to OpenStack and its architecture. It then demonstrates how to quickly get started with Shade by connecting to clouds and listing servers. The document shows how to access OpenStack resources as objects and dictionaries, and generate reports with Pandas. It also discusses implementing Ansible modules with Shade to provision and manage projects across multiple OpenStack clouds. The document encourages contributing to Shade by implementing missing methods and writing functional tests.
Test Drive Deployment with python and nosetest
Test Drive Deployment with python and nosetestRoberto Polli Roberto Polli, a solutions architect at Par-tec, discusses the implementation of a test-driven deployment strategy for legacy infrastructures to facilitate continuous delivery. The document outlines the challenges faced by enterprises in adopting new technologies and emphasizes using Python for testing and documentation in deployment processes. By applying test-driven development (TDD) principles, the case study showcases improvements in LDAP infrastructure management and operational efficiency.
Tox as project descriptor.
Tox as project descriptor.Roberto Polli This document outlines a presentation by Roberto Polli on using Tox as a project descriptor for Python projects, emphasizing its role in continuous integration by managing environments, dependencies, and testing. It discusses isolating Python projects with virtualenv, compares Tox with established Java project standards, and highlights integration with tools like Docker, Jenkins, and Travis CI. The presentation ultimately advocates for the adoption of Tox in Python development to enhance reproducibility and streamline workflows.
Statistics 101 for System Administrators
Statistics 101 for System AdministratorsRoberto Polli The document outlines a presentation by Roberto Polli on using statistics with Python for system administration, focusing on latency issues and data correlation. It covers topics such as using Python libraries for statistical operations, plotting data distributions, and analyzing correlations between datasets. The presentation emphasizes the importance of visualizing data and continuing to collect statistics for better insights.
Will iPython replace bash?
Will iPython replace bash?Roberto Polli The document is a presentation by Roberto Polli discussing the potential for IPython to enhance and complement daily Bash tasks, emphasizing its benefits in terms of speed and reusability without fully replacing Bash. It covers topics such as real-life scenarios for using Python, features of IPython, pitfalls to avoid when using it, and how to script and automate tasks effectively. The conclusion asserts that while IPython is a valuable tool, it is not expected to replace Bash soon.
Pysmbc Python C Modules are Easy
Pysmbc Python C Modules are EasyRoberto Polli This document summarizes a talk given at EuroPython 2012 on contributing to a Python C extension project using GitHub and the Nose testing framework. The talk discusses how Roberto Polli extended the PySmbC library to support extended permissions (ACLs) by wrapping two additional SMB functions into the library and contributing the changes using GitHub. It emphasizes that writing tests with Nose before making code changes makes the contribution process easier. The document provides examples of how functions were wrapped in C and mapped to Python along with how exceptions were defined and handled.
Git gestione comoda del repository
Git gestione comoda del repositoryRoberto Polli Il documento descrive Git, un sistema di controllo versione distribuito, evidenziando la sua utilità nel tracciamento delle modifiche nel codice sorgente. Viene spiegato il processo di creazione di un repository, l'aggiunta di file, e l'uso di branch per gestire lo sviluppo di funzioni in modo isolato. Inoltre, si discutono comandi pratici per le operazioni comuni e vengono forniti esempi su come gestire le modifiche e risolvere eventuali conflitti.
Testing with my sql embedded
Testing with my sql embeddedRoberto Polli The document discusses the integration of MySQL in C programming, focusing on using an embedded server for testing. It covers initialization, connection management, and querying while highlighting thread safety and mutex usage. Additionally, it provides code samples to illustrate a MySQL client application's structure and functionality.
Servizi di messaging & collaboration in mobilità: Il panorama open source
Servizi di messaging & collaboration in mobilità: Il panorama open sourceRoberto Polli Il documento analizza i servizi di messaging e collaborazione mobile, evidenziando l'importanza delle soluzioni open source nel contesto di un mercato mobile dominato da pochi attori. Viene presentata l'esperienza di Babel S.r.l. nello sviluppo della Mailware Collaboration Suite, un prodotto scalabile e interoperabile creato per rispondere alle esigenze di aziende di diverse dimensioni. Inoltre, si sottolinea il ruolo delle comunità open source nel miglioramento continuo del software e nelle innovazioni delle tecnologie utilizzate.
Funambol al Linux Day 2009
Funambol al Linux Day 2009Roberto Polli Il documento discute l'importanza della sincronizzazione dei dati personali (PIM) nell'era dei dispositivi mobili, evidenziando il progetto open source Funambol che implementa il protocollo SyncML. Viene descritta la complessità della sincronizzazione a causa delle differenze nei formati di dati e nei dispositivi, oltre alla necessità di mantenere i dati aggiornati e disponibili su più piattaforme. Si conclude con un esempio pratico di come Funambol gestisce il processo di sincronizzazione tra dispositivi e server.
ICalendar RFC2445 - draft1
ICalendar RFC2445 - draft1Roberto Polli The document discusses calendar standards including iCalendar (RFC2445) which defines a standard format for managing calendar events and tasks. It describes how calendaring allows users to manage their time by recording events, scheduling tasks, and organizing meetings. It also discusses different calendar formats (iCalendar and vCalendar), conversion issues between formats, and CalDAV which provides a standard protocol for accessing calendar events over HTTP.
Presenting CalDAV (draft 1)
Presenting CalDAV (draft 1)Roberto Polli CalDAV is a standard protocol for storing and retrieving calendar events. It is based on WebDAV and HTTP, and allows users to store calendar events in files and folders on a server. Key features include methods for getting, putting, deleting, and querying events; using ETAGs for versioning events; setting access control lists (ACLs) to share calendars; and scheduling capabilities.
Integrating Funambol with CalDAV and LDAP
Integrating Funambol with CalDAV and LDAPRoberto Polli The document discusses integrating Funambol with CalDAV and LDAP to synchronize calendar events across devices. It describes how Babel uses Funambol and CalDAV to integrate calendar syncing with its LDAP authentication system. It also discusses issues with calendar data formats and the Connector Testing Framework used to test calendar syncing functionality.
ultimo-miglio-v3
ultimo-miglio-v3Roberto Polli Il documento discute l'importanza delle reti di telecomunicazione in Italia e le sfide legate all'ultimo miglio, evidenziando come la privatizzazione di Telecom Italia abbia limitato la concorrenza. Introduce il WiMax come soluzione potenziale per connettere aree non servite, permettendo un accesso economico a internet. Infine, analizza le implicazioni dell'assegnazione delle frequenze per WiMax e il rischio di favorire oligopoli a scapito della concorrenza locale.
Ultimo Miglio v2
Ultimo Miglio v2Roberto Polli Il documento discute l'importanza delle reti di telecomunicazioni nella società moderna, evidenziando il problema dell'ultimo miglio e la necessità di una concorrenza reale nel settore. Propone il Wimax come soluzione economica per connettere le case senza adeguata rete telefonica e analizza le problematiche relative all'assegnazione delle frequenze. Infine, si sottolinea il rischio di un'assegnazione a lungo termine delle frequenze che potrebbe favorire oligopolisti a discapito degli operatori locali.
Ultimo Miglio
Ultimo MiglioRoberto Polli Le reti di telecomunicazioni sono cruciali per la società della conoscenza, collegando persone e servizi. In Italia, il problema dell'ultimo miglio limita l'accesso alla rete, rendendo difficile lo sviluppo in alcune aree, mentre il Wimax offre una soluzione economica per superare queste sfide. Le attuali politiche di assegnazione delle frequenze Wimax potrebbero favorire i grandi operatori, penalizzando la concorrenza locale.
Recently uploaded (20)
Vigilanti-Cura-Protecting-the-Faith.pptx
Vigilanti-Cura-Protecting-the-Faith.pptxsecretarysocom On the 29th of June, 1936, Pius XI addressed a pontifical encyclical entitled "Vigilanti Cura" to all the Catholic bishops in the United States. This encyclical was dedicated to "The Motion Picture" and justified his intervention by "the lamentable progress of the motion picture art and industry in the portrayal of sin and vice". I've appropriated the title of his encyclical for my film, putting it to a completely different use to that originally intended. The 'photogenic quality' of the Latin terminology has, however, been preserved and, as per Adorno: "like a neon light which has just been switched on, the commercial and promotional nature of contemporary culture glows brightly". Vigilanti Cura is an irreverent film; insolent and deliberately confusing; a grab-bag of immorality. The montage combines a range of archival imagery from Humanite magazine with images sourced from the media (the military parade of the 14th of July, the current sorry crop of political celebrities ...) or from the cinema (the automatic writing of a puzzle composed of motifs borrowed from existing films). Vigilanti Cura ... or merely a tacit admission of the downfall of contemporary man, drowning in a sea of political, social and religious fundamentalisms. Where is he to be found? Locked in a vis-a-vis with depression and ego. Psychoanalysis no longer serves as a pretext for the dissolution of sexual and social taboos; it now provokes withdrawal into an auto-reflexive isolation, to the extent that societal problems get frozen in the mirror of Auto-Medusification. The mirror has become our idol - let's destroy it!
CBUSDAW - Ash Lewis - Reducing LLM Hallucinations
CBUSDAW - Ash Lewis - Reducing LLM HallucinationsJason Packer Columbus Data & Analytics Wednesdays, June 2025 with Ashley Lewis. A field guide to reducing LLM hallucinations.
最新版美国威斯康星大学绿湾分校毕业证（UWGB毕业证书）原版定制
最新版美国威斯康星大学绿湾分校毕业证（UWGB毕业证书）原版定制Taqyea 鉴于此，定制威斯康星大学绿湾分校学位证书提升履历【q薇1954292140】原版高仿威斯康星大学绿湾分校毕业证(UWGB毕业证书)可先看成品样本【q薇1954292140】帮您解决在美国威斯康星大学绿湾分校未毕业难题，美国毕业证购买，美国文凭购买，【q微1954292140】美国文凭购买，美国文凭定制，美国文凭补办。专业在线定制美国大学文凭，定做美国本科文凭，【q微1954292140】复制美国University of Wisconsin-Green Bay completion letter。在线快速补办美国本科毕业证、硕士文凭证书，购买美国学位证、威斯康星大学绿湾分校Offer，美国大学文凭在线购买。
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
【复刻一套威斯康星大学绿湾分校毕业证成绩单信封等材料最强攻略,Buy University of Wisconsin-Green Bay Transcripts】
购买日韩成绩单、英国大学成绩单、美国大学成绩单、澳洲大学成绩单、加拿大大学成绩单（q微1954292140）新加坡大学成绩单、新西兰大学成绩单、爱尔兰成绩单、西班牙成绩单、德国成绩单。成绩单的意义主要体现在证明学习能力、评估学术背景、展示综合素质、提高录取率，以及是作为留信认证申请材料的一部分。
威斯康星大学绿湾分校成绩单能够体现您的的学习能力，包括威斯康星大学绿湾分校课程成绩、专业能力、研究能力。（q微1954292140）具体来说，成绩报告单通常包含学生的学习技能与习惯、各科成绩以及老师评语等部分，因此，成绩单不仅是学生学术能力的证明，也是评估学生是否适合某个教育项目的重要依据！
The it.com Domains Brand Book for registrars, domain resellers and hosting co...
The it.com Domains Brand Book for registrars, domain resellers and hosting co...it.com Domains We present to you the brand book of it.com Domains. It will help you follow the it.com brand identity guidelines.
Internet & Protocols : A Blueprint of the Internet System
Internet & Protocols : A Blueprint of the Internet Systemcpnabil59 Internet & Protocols: A Blueprint of the Internet System by Nabil Bin Billal is a comprehensive and ambitious guide that unravels the complexity of how the internet works—layer by layer, protocol by protocol, and service by service. This book is designed not just for students or developers, but for anyone curious about the inner structure and functionality of the digital world we depend on every day. Whether you're a beginner hoping to build foundational knowledge or a curious reader aiming to explore deeper layers of internet architecture, this book serves as a solid blueprint.
From the basics of the internet and its protocols to the intricate systems behind servers, databases, hosting, and DNS, the book journeys through every major layer of online connectivity. It explores technologies like TCP/IP, HTTP, DNS, and modern URL systems, helping readers understand how a simple web address leads to a complex interaction between multiple systems and services. With examples from real-life websites such as Apple.com, GitHub Pages, and Bidibo.xyz, concepts are explained clearly and practically.
The book delves into backend processes, covering JavaScript-based interactivity, database integration, and dynamic routing. It then transitions into the world of domains and internet governance, introducing readers to ICANN, registrars, DNS root servers, and the fascinating history of domain name systems. It sheds light on the control structure of the internet, discussing why most digital infrastructure is based in the U.S. and how domain registration has evolved from a free service to a billion-dollar industry.
Security plays a central role in the discussion, with detailed chapters on HTTPS, SSL/TLS, browser verification, and data encryption. It also takes a practical look at performance, introducing concepts like caching, CDNs, and load balancing to show how the modern web achieves speed and stability. The deep and dark web chapters provide insight into the vast unexplored parts of the internet, discussing how anonymity tools like Tor work and what lies beyond the surface web.
A standout feature of the book is its treatment of search engines—both traditional and dark web variants—explaining crawling, indexing, noindex tags, ranking algorithms, and SEO basics. It demystifies how websites appear in Google search results and how metadata plays a key role. Readers are also introduced to the real-time internet, with chapters on IoT, cloud computing, NASA’s deep space networks, Wi-Fi, Bluetooth, Ethernet, and data transmission protocols. Detailed discussions cover IEEE standards, spectrum ranges, SIM card technology, mobile towers, and the very mechanics of electromagnetic communication.
Darley - BSides Nairobi (2025-06-07) Epochalypse 2038 - Time is Not on Our Si...
Darley - BSides Nairobi (2025-06-07) Epochalypse 2038 - Time is Not on Our Si...treyka Epochalypse 2038: Time is Not on Our Side
Presented by Trey Darley, Founder – Threshold Continuity Alliance
BSides Nairobi – 2025-06-07
The Year 2038 Problem is real — and it's already here.
At exactly 03:14:07 UTC on January 19, 2038, 32-bit signed Unix time overflows. Systems that use 32-bit time_t will reset to 1901 and/or fail outright. But this isn’t just about old embedded gear. It’s about trust, cryptographic integrity, log coherence, financial timestamps, system coordination, and the fragile scaffolding of global infrastructure.
This talk explores a dangerous and still largely invisible class of vulnerabilities: timestamp fragility and time synchronization failure.
We cover:
- Why over 60% of global software systems still depend on Unix epoch time
- How 2038 bugs can manifest subtly — without a crash, and without warning
- Why critical infrastructure (energy, telecom, aviation, finance) is especially at risk
- How even modern firmware is being shipped today with latent Y2038 bugs
- The implications for TLS certificates, forensic timelines, billing systems, and safety-critical protocols
- How time can be maliciously spoofed, delayed, or misaligned — and why legacy NTP is often unauthenticated
- A pragmatic call for 2038 rollover testbeds, code audits, and hardened time infrastructure
- Recommendations for deploying secure time protocols (NTS, RFC 8915), GPS + Rubidium/Cesium fallback, and air-gapped sync
You’ll leave with a grounded understanding of:
- Why time isn’t just an input — it’s an untrusted vector
- What engineers, regulators, and defenders must do now to avert a long-tail catastrophe
- How underserved regions may suffer disproportionately — but also leap ahead by refusing to inherit broken time
About the Speaker
Trey Darley is the founder of the Threshold Continuity Alliance (TCA), an initiative focused on strategic risk, infrastructure integrity, and time-based vulnerability remediation. A long-time figure in the global cybersecurity community, Trey works at the intersection of symbolic systems, resilience engineering, and future ethics.
Unlocking Business Growth Through Targeted Social Engagement
Unlocking Business Growth Through Targeted Social EngagementDigital Guider Facebook marketing allows businesses to connect with their ideal audience through precise targeting and engaging content. By leveraging advanced tools like custom audiences, Facebook Ads, and real-time analytics, brands can build strong relationships, increase visibility, and drive measurable results. It’s not just about reaching people—it’s about reaching the right people and turning engagement into growth.
https://digitalguider.com/digital-advertising/facebook-advertising-services/
Communio-et-Progressio - Catholic Church Document on communication
Communio-et-Progressio - Catholic Church Document on communicationsecretarysocom Pontifical Council for Social Communications#Publications
The 22nd February 1992 saw the publication of a pastoral instruction on social communications entitled Aetatis Novae (At the dawn of a new era). This document marked the twentieth anniversary of the Pontifical Commission's very first document, Communio et Progressio (1971). The former document opens with an introduction titled A Revolution in Human Communications. Its body is divided in five parts which include; The Context of Social Communications, The Work of the means of Social Communications, Current Challenges, Pastoral Priorities and Responses, and The need for pastoral planning. Aetatis Novae closes with the conclusion followed by the appendix.
The document Aetatis Novae systematically draws an outline in the propagation of Social Communications media for the pastoral purpose of spreading the Good News of the Lord. It is also based and established from other previous documents of the Catholic Church on social communications media such as the Inter Mirifica and Communio et Progressio. The latter document Aetatis Novae is more like an application of the need for a responsible use of social communications media specifically explained in the ecclesial document Inter Mirifica.
COMPUTER ETHICS AND CRIME.......................................................
COMPUTER ETHICS AND CRIME.......................................................FOOLKUMARI DONT LOOK AT ME
最新版英国北安普顿大学毕业证（UoN毕业证书）原版定制
最新版英国北安普顿大学毕业证（UoN毕业证书）原版定制 Taqyea 挂科无法毕业鉴于此购买文凭【q薇1954292140】一比一原版(UoN毕业证)北安普顿大学毕业证如何办理改成绩单GPA，文凭购买，毕业证办理，文凭办理只是基础业务。【q薇1954292140】一比一还原国外大学毕业证，定制国外大学学历，制作国外大学文凭，复刻国外大学毕业证书。学位证1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。
【办理北安普顿大学成绩单Buy The University of Northampton Transcripts】
购买日韩成绩单、英国大学成绩单、美国大学成绩单、澳洲大学成绩单、加拿大大学成绩单（q微1954292140）新加坡大学成绩单、新西兰大学成绩单、爱尔兰成绩单、西班牙成绩单、德国成绩单。成绩单的意义主要体现在证明学习能力、评估学术背景、展示综合素质、提高录取率，以及是作为留信认证申请材料的一部分。
北安普顿大学成绩单能够体现您的的学习能力，包括北安普顿大学课程成绩、专业能力、研究能力。（q微1954292140）具体来说，成绩报告单通常包含学生的学习技能与习惯、各科成绩以及老师评语等部分，因此，成绩单不仅是学生学术能力的证明，也是评估学生是否适合某个教育项目的重要依据！
Buy The University of Northampton Diploma《正式成绩单论文没过》有文凭却得不到认证。又该怎么办？鉴于此，购买英国毕业证【q微1954292140】英国文凭购买，英国文凭购买，英国文凭定制，英国文凭补办。专业在线定制英国大学文凭，定做英国本科文凭，【q微1954292140】复制英国The University of Northampton completion letter。在线快速补办英国本科毕业证、硕士文凭证书，购买英国学位证、北安普顿大学Offer，英国大学文凭在线购买。
【主营项目】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理毕业证|办理北安普顿大学毕业证（UoN毕业证书）文凭: 买大学毕业证|买大学文凭【q薇1954292140】学位证明书如何办理申请？
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理北安普顿大学毕业证【q薇1954292140】英国学位证（UoN毕业证书）北安普顿大学毕业证书如何办理国外大学毕业证, 文凭办理, 国外文凭办理, 留信网认证
三.材料咨询办理、认证咨询办理北安普顿大学毕业证（UoN毕业证书）请加学历顾问【微信:1954292140】毕业证购买指大学文凭购买，毕业证办理和文凭办理。学院文凭定制，学校原版文凭补办，扫描件文凭定做，100%文凭复刻。经常操作的国家有美国毕业证，英国毕业证，澳洲毕业证，加拿大毕业证，以及德国毕业证，法国毕业证、荷兰毕业证、瑞士毕业证、日本毕业证、韩国毕业证、新西兰毕业证、新加坡毕业证、泰国毕业证、马来西亚毕业证等。包括了本科毕业证，硕士毕业证。
Predicting Site Quality Google Patent US9767157B2 - Behzad Hussain.pdf
Predicting Site Quality Google Patent US9767157B2 - Behzad Hussain.pdfBehzad Hussain 𝐏𝐚𝐭𝐞𝐧𝐭 𝐈𝐧𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧
𝗣𝗮𝘁𝗲𝗻𝘁 𝗡𝗼.: US9767157B2
𝗧𝗶𝘁𝗹𝗲: Predicting Site Quality
𝗔𝘀𝘀𝗶𝗴𝗻𝗲𝗲: Google Inc., Mountain View, CA
𝗜𝗻𝘃𝗲𝗻𝘁𝗼𝗿𝘀: Navneet Panda; Yun Zhou
𝗜𝘀𝘀𝘂𝗲 𝗗𝗮𝘁𝗲: September 19, 2017
𝐀𝐛𝐬𝐭𝐫𝐚𝐜𝐭
This patent describes methods and systems for automatically predicting a quality score for a website (or “site”) that can be used as a ranking signal in search engines:
1. 𝗕𝗮𝘀𝗲𝗹𝗶𝗻𝗲 𝘀𝗶𝘁𝗲 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝘀𝗰𝗼𝗿𝗲𝘀 obtained for previously scored sites.
2. 𝗣𝗵𝗿𝗮𝘀𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 that map phrase-specific relative frequency measures to baseline quality scores.
3. 𝗔𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲 𝘀𝗰𝗼𝗿𝗶𝗻𝗴 of a new site by applying the phrase model to its phrase frequencies and then predicting its overall site quality score from the aggregate.
𝐒𝐲𝐬𝐭𝐞𝐦 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 (𝐅𝐢𝐠. 𝟏)
1. 𝗨𝘀𝗲𝗿 𝗗𝗲𝘃𝗶𝗰𝗲 & 𝗡𝗲𝘁𝘄𝗼𝗿𝗸: Users submit queries via a device (e.g., browser) over a network.
2. 𝗦𝗲𝗮𝗿𝗰𝗵 𝗦𝘆𝘀𝘁𝗲𝗺: Contains an Indexing Engine (builds the index database) and a Ranking Engine (ranks results).
3. 𝗦𝗶𝘁𝗲 𝗦𝗰𝗼𝗿𝗶𝗻𝗴 𝗘𝗻𝗴𝗶𝗻𝗲: Computes site quality scores using the phrase model and provides these scores to the ranking engine as part of the ranking signals.
𝐏𝐡𝐫𝐚𝐬𝐞 𝐌𝐨𝐝𝐞𝐥 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 (𝐅𝐢𝐠. 𝟐)
1. 𝗧𝗼𝗸𝗲𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝗻-𝗴𝗿𝗮𝗺 𝗘𝘅𝘁𝗿𝗮𝗰𝘁𝗶𝗼𝗻: For each site in a corpus of previously scored sites, extract n-grams (typically 2- to 5-grams) from page content.
2. 𝗥𝗲𝗹𝗮𝘁𝗶𝘃𝗲 𝗙𝗿𝗲𝗾𝘂𝗲𝗻𝗰𝘆 𝗖𝗮𝗹𝗰𝘂𝗹𝗮𝘁𝗶𝗼𝗻: For each extracted n-gram, compute its relative frequency as the ratio of pages containing that n-gram to total pages on the site.
3. 𝗕𝘂𝗰𝗸𝗲𝘁 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴: Group sites into 20–100 buckets based on their relative frequency measures for each n-gram, ensuring roughly equal bucket sizes or equal interval ranges.
4. 𝗕𝗮𝘀𝗲𝗹𝗶𝗻𝗲 𝗦𝗰𝗼𝗿𝗲 𝗔𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗶𝗼𝗻: For each bucket, calculate an average baseline quality score from the known scores of sites in that bucket.
5. 𝗣𝗵𝗿𝗮𝘀𝗲 𝗠𝗼𝗱𝗲𝗹 𝗖𝗼𝗻𝘀𝘁𝗿𝘂𝗰𝘁𝗶𝗼𝗻: Associate each n-gram with its vector of bucket-average quality scores. Optionally exclude “neutral” phrases whose scores are statistically indistinguishable from the global average.
𝐒𝐢𝐭𝐞 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 𝐏𝐫𝐞𝐝𝐢𝐜𝐭𝐢𝐨𝐧 𝐏𝐫𝐨𝐜𝐞𝐬𝐬 (𝐅𝐢𝐠. 𝟑)
1. 𝗥𝗲𝗹𝗮𝘁𝗶𝘃𝗲 𝗙𝗿𝗲𝗾𝘂𝗲𝗻𝗰𝘆 𝗳𝗼𝗿 𝗡𝗲𝘄 𝗦𝗶𝘁𝗲: Extract the same set of n-grams from the new (previously unscored) site and compute their relative frequencies.
2. 𝗕𝘂𝗰𝗸𝗲𝘁 𝗟𝗼𝗼𝗸𝘂𝗽: For each phrase, map its relative frequency to the corresponding bucket’s average score in the phrase model.
3. 𝗔𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲 𝗦𝗰𝗼𝗿𝗲 𝗖𝗼𝗺𝗽𝘂𝘁𝗮𝘁𝗶𝗼𝗻: Compute an aggregate score across all phrases’ bucket scores—typically via a weighted or unweighted mean. Weights can reflect phrase frequency, distance from a “neutral” score, or limit the influence of overly frequent phrases.
Common Pitfalls in Magento to Shopify Migration and How to Avoid Them.pdf
Common Pitfalls in Magento to Shopify Migration and How to Avoid Them.pdfCartCoders Planning to migrate from Magento to Shopify? CartCoders offers reliable and error-free Magento to Shopify migration services, ensuring your store data, design, and SEO remain intact. Get expert migration with zero data loss and minimal downtime.
Scaling mysql with python (and Docker).
- 1. Scaling MySQL with python EuroPython 2015, Bilbao Roberto Polli - [email protected] Par-Tec Spa - Rome Operation Unit P.zza S. Benedetto da Norcia, 33 00040, Pomezia RM - www.par-tec.it 20 July 2015 Roberto Polli - [email protected]
- 2. Agenda Intro MySQL Python Utilities Connectors Replication Failover Fabric Provisioning Fabric in the Cloud Roberto Polli - [email protected]
- 3. Who? What? Why? Roberto Polli - Solutions Architect @ par-tec.it. Loves writing in C, Java and Python. Red Hat Certified Engineer and Virtualization Administrator. Par-Tec – Proud sponsor of this talk ;) Contributes to various FLOSS. Provides: • IT Infrastructure & Services expertise; • Business Intelligence solutions; • Vertical Applications for the financial market. Manage, replicate, scale MySQL is easy with python (and Docker). Intro Roberto Polli - [email protected]
- 4. MySQL • Single-process, multi-thread • 2-tier architecture • Connection + SQL • Storage Backend • Greatly improved in 5.6 • More to come in 5.7: Dynamic Buffers & Multi-source replication MySQL Roberto Polli - [email protected]
- 5. MySQL Many Backends aka Storage Engines: • InnoDB1 : ACID, MVCC2 , Redo|Undo logs; • Memory: fast & volatile; • NDB: MySQL Cluster, network distributed, auto-sharded; Replication based on changelog files (binary logs). 1 default 2 Multi Versioning Concurrency Control MySQL Roberto Polli - [email protected]
- 6. Manage & Use a database Monitor: • Database size: Tables, Indexes, Binary Logs • Replication inconsistencies • Failover Connect: • Native Drivers • for Python 2.6, 2.7 and 3.3 • SSL & compression • Orchestrate Download scripts from some blog, add random queries from docs.mysql.com, Shake. MySQL Roberto Polli - [email protected]
- 7. MySQL Utilities A python package with utilities, drivers, fabric orchestrator $ wget http://bit.ly/1CxNuZe $ tar xf mysql-utilities-1.6.1.tar.gz $ cd mysql-utilities-1.6.1 $ python setup.py install mysql-utilities-1.6.1 |-- mysql |-- connector | |-- django | ‘-- fabric |-- fabric | |-- protocols | |-- providers | ‘-- services ‘-- utilities Python Utilities Roberto Polli - [email protected]
- 8. MySQL Utilities: Start with mysqluc for a single entrypoint • entrypoint for all utilities • contextual help • TAB completion Or access each utility separately. Specify server URIs as user:pass@host[:port]. Python Utilities Roberto Polli - [email protected]
- 9. MySQL Utilities mysqluc> help utilities Utility Description ----------------- -------------------------------------------------------- mysqlauditadmin audit log maintenance utility mysqlauditgrep audit log search utility ... mysqldbcompare compare databases for consistency ... mysqldiskusage show disk usage for databases mysqlfailover automatic replication health monitoring and failover Python Utilities Roberto Polli - [email protected]
- 10. Disk Usage A single command to show all disk usage infos (excluded system logs) $ mysqldiskusage --all --server=$SERVER ... | db_name | total | | akonadi | 969150919 | ... Total database disk usage = 971233529 bytes or 926,24 MB ... | ib_logfile0 | 67108864 | | ib_logfile1 | 67108864 | | ibdata1 | 44040192 | Total size of InnoDB files = 178257920 bytes or 170,00 MB Python Utilities Roberto Polli - [email protected]
- 11. Connectors from django.db.backends import BaseDatabaseValidation if django.VERSION < (1, 7): from django.db import models else: from django.core import checks from django.db import connection class DatabaseValidation(BaseDatabaseValidation): if django.VERSION < (1, 7): Connectors Roberto Polli - [email protected]
- 12. Log-based Replication: Master Replication is asynchronous or semi − synchronous. A Master with a Unique Server ID • produces a changelog named binary log; • assign each transaction a Global Transaction ID; • grants access to a replica user; Optionally: • stores replication infos in ACID tables; • may track slave-updates; • waits for one slave ack on semi − synchronous replication; Replication Roberto Polli - [email protected]
- 13. Log-based Replication: Slave Slave • connects to the master as the replica user; • retrieves the binlog; • applies the changes; • ∀ slave ∃! master; Optionally • forbids local changes from non-root users; If binlogs have been purged, you need to import the master database first! Replication Roberto Polli - [email protected]
- 14. Benefits of replication • Availability. • Scaling reads with R/W split. • Scaling reads differencing indexes. • Backup and data warehouse strategies. # mysqlrplshow output # # Replication Topology Graph s-1.docker:3306 (MASTER) | +--- s-3.docker:3306 - (SLAVE) | +--- s-4.docker:3306 - (SLAVE) Replication Roberto Polli - [email protected]
- 15. mysqlreplicate Configuring replication with one command: mysqlreplicate -b --pedantic --master=$MASTER --slave=$SLAVE --rpl-user=repl:rpass # master uuid = 2cda700... # slave uuid = 7520cf0... # Checking for binary logging on master... # Granting replication access to # replication user... # Connecting slave to master... # CHANGE MASTER TO MASTER_HOST = ’172.17.0.5’,... • preliminary checks; • create replica user on the master; • point the slave to the master; • start loading the first available transaction in binlogs; Replication Roberto Polli - [email protected]
- 16. Initialize new slaves Beware! 1. Avoid replicating user provisioning and other host-related setup! @SESSION.SQL_LOG_BIN=0; 2. Binlogs are usually retained for a fixed period - eg 15days. To provision new slave after that period you must: • start from a backup or a snapshot; • apply the binlogs. mysqldbexport >> data.sql --server=root:pass@master --rpl-user=repl:rpass --export=both --rpl=master --all mysqldbimport --server=root:pass@slave data.sql Replication Roberto Polli - [email protected]
- 17. Failover Basics From replication to high availability, automagically discovering slaves. • promote the most updated slave • reconfigure the others • disable the master • eventually fix ip-address # Read carefully the docs of # mysqlfailover and of all # the supported parameters! mysqlfailover --master=$MASTER --discover-slaves-login=root:password --candidates=$SLAVE1,$SLAVE2 --exec-before=/pre-fail.sh --exec-after=/post-fail.sh Failover Roberto Polli - [email protected]
- 18. Fabric Orchestrator You can try the content of those slides downloading code and baked docker images from here: git clone https://github.com/ioggstream/mysql-community cd mysql-community/fabric docker-compose scale fabric=1 slave=4 Let me know after the talk! Fabric Roberto Polli - [email protected]
- 19. Fabric - HLA A python framework for managing, replicating and scaling mysql servers. • aggregate servers in high availability groups • configure single-master replication topologies • monitor and heal failures • director for rw/split • fabric connectors (caching db topology) Fabric Roberto Polli - [email protected]
- 20. Fabric - HLA Fabric Roberto Polli - [email protected]
- 21. mysqlfabric mysqlfabric commands: • manage - setup and manage fabric service • group - administer high availability groups • server - manage, clone, provision and decommission servers • provider - manage server providers (eg. openstack, ...) Further functionalities: • statistics, dump, role, user, • threat, snapshot, event Fabric Roberto Polli - [email protected]
- 22. Manage Setup and administration is done via mysqlfabric manage [setup|start|ping|stop|teardown] Configure in /etc/fabric.cfg • listening ports for xmlrpc service • admin user (eg. fabric to manage servers) • applicative user credentials • db to store fabric data Setup and start the service mysqlfabric manage setup mysqlfabric manage start --daemon Fabric Roberto Polli - [email protected]
- 23. Replication Groups Groups are managed via mysqlfabric group [create|add|promote|activate] $GROUP Crete an High Availability group and add a bunch of servers. mysqlfabric group create $GROUP mysqlfabric group add $GROUP my-host-1:3306 mysqlfabric group add $GROUP my-host-2:3306 mysqlfabric group add $GROUP my-host-3:3306 Fabric Roberto Polli - [email protected]
- 24. Replication Groups Promote one server as a master, eventually picking one. mysqlfabric group promote $GROUP [--slave_id=UUID] Show the server group mysqlfabric group lookup_servers ha Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e Time-To-Live: 1 server_uuid address status mode weight ------------ ----------- --------- --------- ------ 2cda..110015 172.17.0.21 PRIMARY READ_WRITE 1.0 2cda..110016 172.17.0.22 SECONDARY READ_ONLY 1.0 2cda..110017 172.17.0.23 SECONDARY READ_ONLY 1.0 Fabric Roberto Polli - [email protected]
- 25. Replication Groups Fabric support spare servers. mysqlfabric server set_status $UUID spare Monitoring failover and deactivating the master. mysqlfabric group activate $GROUP mysqladmin -h $MASTER shutdown Checking group health we’ll see that... mysqlfabric group health ha uuid is_alive status ..error status..io_error sql_error ----------- -------- --------- --- --- --- --- -------- --------- 2cd..110015 0 FAULTY 0 0 0 0 False False 2cd..110016 1 SECONDARY 0 0 0 0 False False 2cd..110017 1 PRIMARY 0 0 0 0 False False Fabric Roberto Polli - [email protected]
- 26. Connecting programmatically from mysql.connector import connect, fabric # Ask for a connection to the fabric server. c = connect(fabric={host: .., port: .., user: ..}, autocommit=True, database=’sample’, **sql_user) # Fabric will point you to a suitable host # - in this case the master - of the given # group c.set_property(mode=fabric.MODE_READWRITE, group="my-cluster-1") Fabric Roberto Polli - [email protected]
- 27. Provisioning a new slave Fabric uses mysqldump + mysql internally to clone a new slave. # Always reset TARGET configuration # before reinitializing mysql -e ’ SET @SESSION.SQL_LOG_BIN=0; STOP SLAVE; RESET MASTER; ’ Cloning doesn’t attach the server to a group, nor starts the replica. mysqlfabric server clone $GROUP $TARGET Provisioning Roberto Polli - [email protected]
- 28. Reingesting a failed master Reingesting a failed master is not trivial. • non-replicated transactions; • corrupt data; Always reset it! mysqlfabric server set_status f484...110039 spare mysqlfabric server set_status f484...110039 secondary Provisioning Roberto Polli - [email protected]
- 29. Fabric in the Cloud Fabric can provision new machines via eg. Openstack API. We implemented a DockerProvider: deploy containers, not machines. # mysql.fabric.providers.dockerprovider class MachineManager(AbstractMachineManager): """Manage Docker Containers.""" def create(self, parameters, wait_spawning): ... def search(self, generic_filters, meta_filters): ... def destroy(self, machine_uuid): ... Fabric in the Cloud Roberto Polli - [email protected]
- 30. Docker Provisioning # Register a provider (requires docker in http) mysqlfabric provider register mydocker user password http://172.17.42.1:2375 --provider_type=DOCKER # Create or remove Containers mysqlfabric server create mydocker --image=name=ioggstream/mysql-community --flavor=name=v1 --meta=command="mysqld --log-bin=foo" # List servers mysqlfabric server list docker Fabric in the Cloud Roberto Polli - [email protected]
- 31. Next steps Openstack interface supports machine snapshots via NovaClient. Docker volumes doesn’t support snapshots, but something is moving (in the experimental tree). VM Snapshot alternatives require root access to docker host and a FLUSH TABLES on the source container: • rsync on volumes • implement snapshot via docker volume plugins Fabric in the Cloud Roberto Polli - [email protected]
- 32. Wrap Up • Replication is easier with Fabric and MySQL 5.6 • You can clone servers • Failover is just one command ahead • Don’t re-ingest failed masters (luckily;) • Try Fabric with Docker! • Play with docker volumes Fabric in the Cloud Roberto Polli - [email protected]
- 33. That’s all folks! Thank you for the attention! Roberto Polli - [email protected] Fabric in the Cloud Roberto Polli - [email protected]