Python on exadata

The Pythonic Way crosses
Oracle‘s Exadata
by rainer schuettengruber

about me
• IT employee since 1998
• main focus on Oracle databases
• various positions as Oracle DBA(DMA), developer, devops
• currently employed as Exadata Administrator

continous integration
• version control
• automated build
• unit tests
• automated unit tests
• automated deployment

pre python era
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• heterogeneous environment
• Oracle on AIX, Linux on VMware, Exadata
• Oracle Version 10.2.0.4, 11.2.0.1,11.2.0.2, 11.2.0.3, 11.2.0.4
• Oracle Cloud Control 11g on Linux
• Oracle Cloud Control 12c on AIX
• backup based on Cloud Control jobs
• backup based on TSM scheduler
• backup based on ksh scripts
• .. to take arms against a sea of troubles ..

pre python era
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• automation and standardisation are siamese twins
• standardisation by migrating all databases to Exadata
• migrated on database after the other
• took almost a year
• set the stage for automation

the game is on
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
oracle@disguised:misc (TEST1) > ./tracemaint.sh -h
usage: tracemaint [-h] -d DAYS -m MAXLOGSIZE [-s SID] [-k] [--debug]
tidy up database trace and log files
optional arguments:
-h, --help show this help message and exit
-d DAYS, --days DAYS number of days that logs/traces need to be kept
-m MAXLOGSIZE, --maxlogsize MAXLOGSIZE
threshold in MB, specifying if a log files needs
rotation
-s SID, --sid SID database SID, if omitted all databases configured in
/etc/oratab are considered
-k, --keeptempfiles keep files generated during logrotation, use this for
debugging purposes only
--debug log debug output

the game is on
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• Python comes with Linux, however
• versions, depending on the distribution, between 2.4 – 2.7
• changing installation might do harm, especially on Exadata
• dedicated installation under /opt in accord with FHS
• moreover Exadata does not support customized RPM‘s
• building from source and installation via tarball

the game is on
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
disguised:/root/svnRepo/scripts/bash/osSetup>./build_python.sh
usage : build_python.sh -p <python source tarball> -r -c
-r .. remove existing installation
-c .. create tarball from the installation
disguised:/root/svnRepo/scripts/bash/osSetup>
disguised:/root/svnRepo/scripts/bash/osSetup>./install_python.sh
usage : install_python.sh -t <python tarball>
disguised:/root/svnRepo/scripts/bash/osSetup>

the game is on
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• SVN rather obvious
• used for deployments in combination with make files
• installation under /opt requires command line wrappers
• which comes in handy for cx_Oracle

the game is on
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
[root@disguised traceMaintenance]# cat Makefile
INSTALL_DIR = /home/oracle/scripts/misc
wrapperScripts = $(INSTALL_DIR)/tracemaint.sh
all: $(wrapperScripts)
$(INSTALL_DIR)/%.sh: %.sh
install -o oracle -g oinstall -m 755 $? $@
[root@disguised traceMaintenance]# cat tracemaint.sh
#!/bin/bash
export LD_LIBRARY_PATH=/opt/oracle/product/12.2.0.1/instantclient
unset NLS_LANG
/opt/python/bin/tracemaint "$@"
[root@disguised traceMaintenance]#

the game is on
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
import logging
from logging.handlers import RotatingFileHandler
import argparse
import sys
import os
import subprocess
import glob
import re
from datetime import datetime
import cx_Oracle

OOP appears on the stage
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• SAP requires a dedicated toolset for database maintenance
• basically 4 compenents
• separate for each database
• installation and patching rather tedious for 20+ databases
• implementation reminiscent of the DRY principle

07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• one python file per class, used modules without recognizing it
• patching became a matter of minutes as opposed to hours
• off the beaten script track
• peer cluster nodes taken into consideration

07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
import shutil
from pwd import getpwuid
from pexpect import pxssh
import paramiko
from paramiko import SSHClient
from scp import SCPClient

the master‘s class
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• became aware of python‘s modul concept
• formed the idea of bundling modules in a package that provides console
scripts
• amongst improving python skills, set the stage for a clean and reusable code
base
• good riddance bash
• well established principles in software development won‘t do harm in
operations

the helping hand
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• Cloud Control migration/consilidation had become inevitable
• upgrade path for only one of the two systems available
• implies that 50 per cent of the monitored databases need to be configured
manually, tedious and prone to error
• further use of configuration scripts justify the effort and contribute to
standardisation
• in essence a wrapper for Cloud Control‘s python based emcli utility
• migration/consolidation a matter of hours
• adding a database became a matter of seconds

exadata migration on steriods
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• existing Exadata systems had done their duty
• essentially the same migration path as for the migration from AIX to Exadata
• however, due to strigent licence terms only a parallel phase of 3 months in
sharp contrast to 1 year
• further cut down to 4 weekends due to staff availability
• no reason to despair since python is around ...
• built on top of Oracle‘s dataguard, cutting down downtime to a couple of
minutes

07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• typed python code instead of oracle commands on the weekends
• structured code into modules
• created a package to ease deployment
• manual build with python setup.py bdist_wheel
• implemented console scripts
• toyed with the idea to use ant as build tool

07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
disguised:/root/svnRepo/scripts/python>find rlb -maxdepth 1
rlb
rlb/common
rlb/__init__.py
rlb/__pycache__
rlb/occ
rlb/sap
rlb/tsm
rlb/db
rlb/dg
rlb/dp
rlb/occut
rlb/dput
rlb/commonut
rlb/exa
rlb/tmut
rlb/dbut
rlb/dgut
rlb/tm

07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
from setuptools import setup
from setuptools import find_packages
setup(
name='rlb',
version='6.8.1',
author='rainer schuettengruber',
author_email='rainer.schuettengruber@rrz.co.at',
packages=find_packages(),
package_data={
'': ['*.template'],
},
entry_points={
'console_scripts': [
'massdeploy=rlb.common.massdeploy:main_for_script',
'tracemaint=rlb.tm.tracemaint:main_for_script',

automatisation of the automatisation
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• automated database setup
• build and deployment appeared to be rather tedious
• opted for ant
• faced the same issues as already discussed for the pyhton installation
• addressed by installing ant under /opt
• build specific environment variables by means of the build.env script

automatisation of the automatisation
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
[root@disguised python]# source build.env
PATH :
/opt/python/bin:/opt/ant/bin:/u01/app/12.1.0.2/grid/bin:/usr/local/sbin:/usr/local/bin:/sbi
n:/bin:/usr/sbin:/usr/bin:/root/bin
LD_LIBRARY_PATH : /opt/oracle/product/12.2.0.1/instantclient
ANT_ARGS : -emacs -logger org.apache.tools.ant.NoBannerLogger
PYTHON_PATH : /root/svnRepo/scripts/python
[root@disguised python]# ant -p
Buildfile: /root/svnRepo/scripts/python/build.xml
Build and deploy rlb's pyhton modules
Main targets:
clean clean up all files created by either dist or docs target
dist build wheel
distclean clean up files created by the dist target
install install wheel on this host
Default target: usage
[root@disguised python]#

prose meets code
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• given a SLOC of 3600, documentation became a necessisty
• opted for sphinx
• created UML diagrams with pyreverse which is part of the pylint package
• added ant target docs

prose meets code
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
<target name="docs"
description="generate sphinx documentation, including pylint and pyreverse
output"
depends="clean">

<exec executable="${python.binpath}/sphinx-apidoc">
<arg value="-f" />
<arg value="-o" />
<arg value="${doc.sphinx.source.dir}" />
<arg value="${package.dir}" />
</exec>

<exec executable="${python}" >
<arg value="builduml.py" />
</exec>

<exec executable="/usr/bin/make">
<env key="LD_LIBRARY_PATH" path="/opt/oracle/product/12.2.0.1/instantclient" />
<arg value="html" />
</exec>

prose meets code
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

<mkdir dir="${doc.pylint.dir}" />
<echo message="writing pylint's output to ${doc.pylint.dir}/packages_pylint.txt" />
<exec executable="${pylint}"
dir="${package.dir}"
output="${doc.pylint.dir}/packages_pylint.txt" >
<arg value="--rcfile=pylint.rc" />
<arg value="-f" />
<arg value="text" />
<arg value="rlb" />
</exec>

<delete>
<fileset dir="${doc.deploy.dir}" includes="**/*.*" />
</delete>
<echo message="deploying docu to ${doc.deploy.dir} ..." />
<copy todir="${doc.deploy.dir}">
<fileset dir="${doc.sphinx.build.dir}/html">
<include name="**/*" />
</fileset>
</copy>
</target>

prose meets code
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

a giant leap towards CI
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• functionality shared by all modules has been refactored into the common sub
package
• breaking functionality within the common sub package results in breaking the
whole code base
• unit tests absolutely vital
• additional sub package with suffix ut
• one test file for each module, prefixed with test_

07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
[root@disguised rlb]# find common* -name '*.py'
common/os.py
common/template.py
common/__init__.py
common/oradb.py
common/massdeploy.py
commonut/test_oradb.py
commonut/__init__.py
commonut/test_template.py
commonut/test_os.py
commonut/test_massdeploy.py
[root@disguised rlb]#

07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• version control, ant build and unit tests have been implemented
• which sets the stage for a decent jenkins installation
• Subversion plugin for obvious reasons
• Cobertura plugin aimed at automatic unit tests
• SLOCCount plugin in case somebody asks
• Violations plugin since pylint reports are already in place
• Static analysis collector plugin-in as a dependency

07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
disguised:/root/svnRepo/scripts/python>cat requirements_test.txt
astroid>=1.5.3
autopep8>=1.3.2
coverage>=4.4.1
idna>=2.1
isort>=4.2.5
lazy_object_proxy>=1.2.2
logilab_common>=1.3.0
mccabe>=0.5.2
pep8>=1.7.0
py>=1.4.34
pyenchant>=1.6.8
pygount>=0.9
pylint>=1.7.2
pytest>=3.2.1
pytest-cov>=2.5.1
sphinx_pyreverse>=0.0.12
wrapt>=1.10.8
Pygments>=2.2

07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
<target name="code-metrics" description="gather code metrics">

<echo message="writing pygount's output to cloc.xml" />
<exec executable="${pygount}" dir="../..">
<arg value="--format=cloc-xml" />
<arg value="--suffix=py,sh,sql,xml,pks,pkb,php,java" />
<arg value="--out=cloc.xml" />
</exec>

.
.

<echo message="writing pep8's output to pep8.out" />
<exec executable="${pep8}"
dir="${package.dir}"
output="../../pep8.out">
<arg value="--filename=*.py" />
<arg value="--max-line-length=99" />
<arg value="." />
</exec>
</target>

07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
disguised:/root/svnRepo/scripts/python>ant -p
Main targets:
all build wheel and documentation
code-metrics gather code metrics
dist build wheel
docs generate sphinx documentation, including pylint and pyreverse output
jenkins build targets required by continuous integration
disguised:/root/svnRepo/scripts/python>
<target name="jenkins" description="build targets required by continuous integration"
depends="clean, code-metrics, dist" />

07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

a bit quality
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• given that jenkins and unit tests are in place, test automatisation appears to
be quite obvious
• isolated build by means of venv
• documented required modules, which comes in handy when
upgrading/installing python
• run test suite by means of pytest
• added coverage reports

a bit quality
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
<target name="test" description="run test suite" depends="venv">
<exec executable="${venv.python}" failonerror="true">
<arg value="setup.py" />
<arg value="install" />
</exec>
<exec executable="/home/jenkins/scripts/pre_unittest.sh" />
<exec executable="${venv.pytest}">

<env key="LD_LIBRARY_PATH" path="/opt/oracle/product/12.2.0.1/instantclient" />
<arg value="--junit-xml=unittest.xml" />
<arg value="--cov=${package.dir}" />
<arg value="--cov-report=html" />
<arg value="--cov-report=xml" />
<arg value="--ignore=${venv.dir}" />
<arg value="--ignore=./rlb/commonut/test_cp1252_encoding.txt" />
<arg file="${package.dir}" />
</exec>
<exec executable="/home/jenkins/scripts/post_unittest.sh" />
</target>

a bit quality
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

a bit quality
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• complete test coverage would imply a dedicated exadata environment
• a case for unittest.mock
• as a precondition Exadata/database related calls need to be separated, which
has not been considered in the initial design
• might be subject to another talk
• however, the foundations are laid
• played a vital role during the upgrade vom python 3.5 to 3.6

the finishing touch
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• due to stringent security policies no access to pypi.python.org
• has been circumvented by using pip install http://...
• however, resolving dependencies appeared to be a bit tedious
• so pypiserver it is
• enables continous deployment

the finishing touch
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
[Unit]
Description=pypi-server
After=network.target
[Service]
Type=simple
ExecStart=/opt/python/bin/pypi-server -p 8081 --disable-fallback -v --log-file
/var/log/pypi-server.log -u /var/www/html/pypi
[Install]
WantedBy=multi-user.target

the finishing touch
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
disguised:/root/svnRepo/scripts/python>ant -p
Main targets:
all build wheel and documentation
code-metrics gather code metrics
deploy install wheel on production hosts
dist build wheel
docs generate sphinx documentation, including pylint and pyreverse output
jenkins build targets required by continuous integration
test run test suite
venv setup virtual Python environment
disguised:/root/svnRepo/scripts/python>

where do we go now?
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17
• deal with testing issues
• become proactive by forecasting capacity and load with appropriate models
• tweak pylint/pep8
• improve on documentation

questions and answers
07/14
07/15
09/15
02/16
05/16
07/16
10/16
11/16
01/17
03/17
07/17

Python on exadata

More Related Content

What's hot (20)

Similar to Python on exadata (20)

Recently uploaded (20)

Python on exadata