Open source projects with python
4 likes1,675 views
The document outlines steps to create an open source project using Python, specifically detailing the project 'dividemo,' a command line tool for dividing integers. It covers key aspects such as project setup, version management, testing, continuous integration, and documentation tools like Sphinx and Read the Docs. The guide is targeted at developers interested in starting or enhancing Python open source projects, emphasizing best practices and available tools for efficiency and collaboration.
![Dividemo
Command line tool to divide two integer numbers
and print the result to the standard output:
$ python dividemo.py 8 2
4
$ python dividemo.py 11 3
3
$ python dividemo.py 11 three
usage: dividemo.py [-h] DIVIDEND DIVISOR
dividemo.py: error: argument DIVISOR: invalid int value: 'three'](https://image.slidesharecdn.com/opensourceprojectswithpython-150425081901-conversion-gate02/85/Open-source-projects-with-python-10-320.jpg)
![Source Code
import argparse
import math
import sys
def divided(dividend, divisor):
return dividend // divisor # //=integer division
def main(arguments):
# Parse command line arguments.
parser = argparse.ArgumentParser(description='divide two integer numbers.')
parser.add_argument('dividend', metavar='DIVIDEND', type=int, help='number to divide')
parser.add_argument('divisor', metavar='DIVISOR', type=int,help='number to divide by')
args = parser.parse_args(arguments)
# Process arguments and print result.
result = divided(args.dividend, args.divisor)
print(result)
if __name__ == '__main__':
main(sys.argv[1:])](https://image.slidesharecdn.com/opensourceprojectswithpython-150425081901-conversion-gate02/85/Open-source-projects-with-python-11-320.jpg)
![A test program
import unittest
from dividemo import dividemo
class DividemoTest(unittest.TestCase):
def test_can_divide(self):
self.assertEqual(2, dividemo.divided(10, 5))
def test_can_print_divided(self):
dividemo.main(['10', '5'])
def test_fails_on_non_integer_divisor(self):
self.assertRaises(SystemExit, dividemo.main, ['10', 'hello'])](https://image.slidesharecdn.com/opensourceprojectswithpython-150425081901-conversion-gate02/85/Open-source-projects-with-python-32-320.jpg)
![.travis.yml
language: python
sudo: true
virtualenv:
system_site_packages: true
env:
matrix:
- DISTRIB="ubuntu" PYTHON_VERSION="2.7" COVERAGE="true"
- DISTRIB="conda" PYTHON_VERSION="2.7" COVERAGE="false"
- DISTRIB="conda" PYTHON_VERSION="3.3" COVERAGE="false"
- DISTRIB="conda" PYTHON_VERSION="3.4" COVERAGE="false"
install:
- source tests/travis_install.sh
- pip install -r requirements.txt
before_script:
- git config --global user.email "roskakori@users.sourceforge.net"
- git config --global user.name "Thomas Aglassinger"
script:
- python setup.py test
after_success:
- if [[ "$COVERAGE" == "true" ]]; then coveralls || echo "failed"; fi
cache:
- apt](https://image.slidesharecdn.com/opensourceprojectswithpython-150425081901-conversion-gate02/85/Open-source-projects-with-python-35-320.jpg)
![Version numbering with Pyscaffold
“Magic” in _version.py:
$ python
>>> from dividemo import _version
>>> _version.get_version()['version']
'0.0.post0.dev2+g8cdc4ea'](https://image.slidesharecdn.com/opensourceprojectswithpython-150425081901-conversion-gate02/85/Open-source-projects-with-python-41-320.jpg)
![Launcher scripts
● To just run “dividemo” instead of “python ...”
● Add pointer to main() function in setup.cfg:
[console_scripts]
dividemo = dividemo.dividemo:main](https://image.slidesharecdn.com/opensourceprojectswithpython-150425081901-conversion-gate02/85/Open-source-projects-with-python-59-320.jpg)
![PyPI first time setup
● Prepare ~/.pypirc:
● Register you project:
[distutils]
index-servers=pypi
[pypi]
repository=https://pypi.python.org/pypi
username=roskakori
password=.....
$ python setup.py register](https://image.slidesharecdn.com/opensourceprojectswithpython-150425081901-conversion-gate02/85/Open-source-projects-with-python-61-320.jpg)
Open source projects with python
- 1. Open source projects with Python Thomas Aglassinger http://roskakori.at https://github.com/roskakori/talks/linuxtage @TAglassinger Version 1.0.1
- 2. About me ● 1995: first open source commit ● 2001: master's degree in information processing science (Oulu, Finland) ● 2001: Systema Human Information Systems (Steyr, Austria) ● 2004: Raiffeisen Rechenzentrum (Graz, Austria) ● “Casual” open source developer; Open Hub commit history:
- 3. Goals ● Beginning from an empty folder a small open source project is published. ● The project uses Python as programming language. ● The project is easy to setup and maintain. ● The project is easy to contribute to. ● The project uses quality oriented approach and utilizes modern tools and best practices.
- 4. Intended audience ● Developers who intend to start an open source project ● People who are already working on open source Projects in Python ● Anyone who wants to understand the processes and tools in the background of an open source project
- 5. Topics ● Naming and licensing a project ● The sample project: dividemo ● Version management ● Project structure ● Writing a REAME ● Build process ● Testing and continuous integration ● Version numbering ● Static code checks ● Merging contributions ● Documentation ● Launcher scripts ● Publishing
- 7. Naming a project ● No http://<project>.com ● No clashes with existing open source projects: https://github.com/LogIN-/ospnc ● Other considerations: https://www.farbeyondcode.com/Choosing-a-name-for-your-open-source-project--5-2700.html
- 8. Licensing ● Many licenses to choose from: http://opensource.org/licenses ● Basic choices: http://choosealicense.com/ ● Popular: – GNU General Public License (GPL) – GNU Library General Public License (LGPL) – Apache License – MIT License – BSD License ● For our simple example program: BSD License
- 10. Dividemo Command line tool to divide two integer numbers and print the result to the standard output: $ python dividemo.py 8 2 4 $ python dividemo.py 11 3 3 $ python dividemo.py 11 three usage: dividemo.py [-h] DIVIDEND DIVISOR dividemo.py: error: argument DIVISOR: invalid int value: 'three'
- 11. Source Code import argparse import math import sys def divided(dividend, divisor): return dividend // divisor # //=integer division def main(arguments): # Parse command line arguments. parser = argparse.ArgumentParser(description='divide two integer numbers.') parser.add_argument('dividend', metavar='DIVIDEND', type=int, help='number to divide') parser.add_argument('divisor', metavar='DIVISOR', type=int,help='number to divide by') args = parser.parse_args(arguments) # Process arguments and print result. result = divided(args.dividend, args.divisor) print(result) if __name__ == '__main__': main(sys.argv[1:])
- 12. Store the source code $ cd ~/workspace # (or something the like) $ mkdir --parents dividemo/dividemo $ $EDITOR dividemo/dividemo/dividemo.py
- 14. Version management - goals ● Changes can be tracked ● Other people can easily contribute ● Improvements can be merged easily ● Mistakes in the code can be undone by reverting to a working version
- 15. Popular services ● https://github.com ● https://bitbucket.com ● http://sourceforge.net
- 16. Create a project on Github
- 17. Project repository URL ● Depend on username and project name ● https://github.com/roskakori/dividemo.git
- 19. Project structure ● “5 Simple Rules For Building Great Python Packages”: http://axialcorps.com/2013/08/29/5-simple-rules-for-building-great-python-packages/ ● Pyscaffold can help to putup a new project from scratch: https://pypi.python.org/pypi/pyscaffold
- 20. Pyscaffold ● Builds a scaffold for a new project ● Preconfigures helpful utilities $ cd ~/workspace # (or something the like) $ putup --description "divide two integer numbers" --url https://github.com/roskakori/dividemo --license "simple-bsd" --with-travis --with-tox dividemo --force
- 21. Connect with Github $ git remote add origin https://github.com/roskakori/dividemo.git $ git push -u origin master To avoid entering the Github password on each push: https://help.github.com/articles/generating-ssh-keys/
- 23. Writing a README ● Should contain: – Short summary of use – Possibly a concise example – Reference to license, documentation, source code – Where to get support? (email, issue tracker) ● For small applications: README = documentation ● Format: ReStructured Text or Markdown ● Online editor: http://rst.ninjs.org/
- 24. README.rst Dividemo ======== Dividemo is a command line tool that divides two integer numbers and prints the result in the console. Example:: $ dividemo 11 4 2 For more information, visit https://dividemo.readthedocs.org. To get support, open an issue at https://github.com/roskakori/dividemo/issues. The source code is available from https://github.com/roskakori/dividemo. License ------- Copyright (c) 2015, Thomas Aglassinger. Distributed under the BSD license. See LICENSE.txt for more information.
- 25. Build process
- 26. Setup.py (1/2) ● Acts both as – Installer – Build tool ● Supported by standard library (distutil) ● Most things are are simple declarations (e.g. license, author, dependencies) ● If necessary, all functions of the Python library can be used → no limited “scripting language” like make, ant, ...
- 27. Setup.py (2/2) ● Has several built in commands, e.g. build and install the package ● External packages can add additional commands (e.g. pip, wheel) ● You can also add your own commands → pure Python code directly in setup.py
- 28. Build package ● python setup.py sdist build source distribution as *.tar.gz ● python setup.py sdist --formats=zip build source distribution as *.zip ● Python setup.by bdist_wheel build binary distribution as wheel; requires wheel package, see https://pypi.python.org/pypi/wheel
- 29. requirements.txt ● Describes dependencies to other Python packages that need to be installed ● Simple syntax; one line per package ● Example: requests >= 2.6.2 → requires requests package, version 2.6.2 or later ● Pip automatically installs packages described in requirements.txt
- 30. Installation ● python setup.py develop “Install” current development source code (make it part of PYTHONPATH) ● python setup.py install Install package in current virtualenv, see https://pypi.python.org/pypi/virtualenv ● sudo python setup.py install Install package in system folders
- 31. Testing
- 32. A test program import unittest from dividemo import dividemo class DividemoTest(unittest.TestCase): def test_can_divide(self): self.assertEqual(2, dividemo.divided(10, 5)) def test_can_print_divided(self): dividemo.main(['10', '5']) def test_fails_on_non_integer_divisor(self): self.assertRaises(SystemExit, dividemo.main, ['10', 'hello'])
- 33. Run test suite ● Requires configuration or automatic configuration by PyScaffold ● python setup.py test Runs test suite, reports result and builds HTML report about test coverage ● Coverage reports is located in folder “htmlcov”.
- 34. Continuous integration ● After each push to the version management repository, run the test suite → make you aware of new bugs early ● Travis - https://travis-ci.org/ – Github informs Travis about new push – Travis runs tests – If tests fail, Travis sends e-mail – Test log is available online ● Jenkins - http://jenkins-ci.org/ – Can be deployed locally – Python setup: http://www.alexconrad.org/2011/10/jenkins-and-python.html
- 35. .travis.yml language: python sudo: true virtualenv: system_site_packages: true env: matrix: - DISTRIB="ubuntu" PYTHON_VERSION="2.7" COVERAGE="true" - DISTRIB="conda" PYTHON_VERSION="2.7" COVERAGE="false" - DISTRIB="conda" PYTHON_VERSION="3.3" COVERAGE="false" - DISTRIB="conda" PYTHON_VERSION="3.4" COVERAGE="false" install: - source tests/travis_install.sh - pip install -r requirements.txt before_script: - git config --global user.email "[email protected]" - git config --global user.name "Thomas Aglassinger" script: - python setup.py test after_success: - if [[ "$COVERAGE" == "true" ]]; then coveralls || echo "failed"; fi cache: - apt
- 36. Activate travis ● Visit https://travis-ci.org/profile ● Click ● Enable project:
- 37. Coveralls ● Online test coverage reports ● Add a repository: https://coveralls.io/repos/new ● (even for open source repos) ●
- 38. Time to git push!
- 40. Pythoner version numbering ● Guidelines: “PEP 440 - Version Identification and Dependency Specification” https://www.python.org/dev/peps/pep-0440/ ● Easy but cumbersome: manual maintenance in __init__.py: __version__ = '1.2.3'
- 41. Version numbering with Pyscaffold “Magic” in _version.py: $ python >>> from dividemo import _version >>> _version.get_version()['version'] '0.0.post0.dev2+g8cdc4ea'
- 42. Advancing the version Add a new git tag: $ git tag -a -m "Tagged version 0.1.0." v0.1.0 $ git push --tags
- 43. Trove classifiers ● Describe package ● Make it easier for users to find it ● Available classifiers: https://pypi.python.org/pypi?%3Aaction=list_classifiers
- 44. Add trove classifiers ● Setup.py ● With Pyscaffold: setup.cfg
- 45. Dividemo trove classifiers classifiers = Development Status :: 4 - Beta, Environment :: Console, License :: OSI Approved :: BSD License, Operating System :: OS Independent, Programming Language :: Python, Programming Language :: Python :: 2, Programming Language :: Python :: 3, Topic :: Utilities
- 46. Meanwhile...
- 47. Travis finished
- 49. Static code checks ● Identify possibly issues by scanning the source code ● PEP8 Style guide for Python code http://legacy.python.org/dev/peps/pep-0008/ ● “Code smells”, e.g. unreachable or unused code ● Intended to improve general code quality and simplify maintenance
- 50. flake8 ● Finds formatting issues and a few code smells ● Pragmatic and low volume $ tox -e flake8 dividemo/dividemo.py:5:1: F401 'math' imported but unused dividemo/dividemo.py:11:1: E302 expected 2 blank lines, found 1 dividemo/dividemo.py:14:80: E501 line too long (90 > 79 characters) dividemo/dividemo.py:15:64: E231 missing whitespace after ',' ...
- 51. Pylint ● http://www.pylint.org/ ● Provides many checks ● Default setting: very verbose, lots of noise ● Simple front end: https://landscape.io/ ● Based on prospector https://pypi.python.org/pypi/prospector
- 53. Pull requests ● Feature of Github and Bitbucket ● Makes it easy to review, iterate and merge changes from another fork ● Sadly no live presentation due lack of time ggg:-(
- 54. Documentation
- 55. Sphinx documentation ● “Sphinx is a tool that makes it easy to create intelligent and beautiful documentation” ● http://sphinx-doc.org ● Based on ReStructured Text markup ● Easy linking and cross referencing ● Automatically builds index and search page ● Extract API documentation from source code
- 56. Sphinx configuration ● docs/conf.py ● Manually: docs/Makefile ● Pyscaffold: ● Possibly have to set theme in conf.py: html_theme = 'default' ● Possibly trim option intersphinx_mapping ● HTML results are located in “docs/_build/html” $ python setup.py docs
- 57. Publishing the documentation ● https://readthedocs.org ● After push to repository, rebuild and publish the documentation ● Dashboard > Import a project > From Github ● Wait for build to finish ● Read it at https://dividemo.readthedocs.org
- 58. Launcher script
- 59. Launcher scripts ● To just run “dividemo” instead of “python ...” ● Add pointer to main() function in setup.cfg: [console_scripts] dividemo = dividemo.dividemo:main
- 60. Publish to the Python Package Index (PyPI)
- 61. PyPI first time setup ● Prepare ~/.pypirc: ● Register you project: [distutils] index-servers=pypi [pypi] repository=https://pypi.python.org/pypi username=roskakori password=..... $ python setup.py register
- 62. Publish a new version $ git tag -a -m "Tagged version 1.0.0." v1.0.0 $ git push –tags $ python setup.py sdist --formats=zip upload
- 63. Summary
- 64. Summary ● Many helpful tools for Python projects – Flake8, git, pip, Pyscaffold, sphinx ● Many helpful services – Coveralls, Github, PyPI, Readthedocs, Travis ● Easy collaboration with Github and Pull requests ● Python is fun!