Python packaging and dependency resolution
0 likes1,346 views
This document discusses Python packaging and improving dependency resolution. It provides an overview of packaging, including creating packages with setup.py and uploading them to a package server. It then discusses challenges with early packaging tools like Distutils and improvements with setuptools, pip, and virtualenv. It also examines how pip handles dependency inconsistencies and the importance of pinning dependencies precisely in requirements.txt. Finally, it recommends hosting your own private package index or proxy to improve reliability.
![Packaging creation upload
# ~/.pypirc
[company]
username: Andreia
password: pwd
repository: http://pypi.company.com
$ python setup.py sdist upload -r company](https://image.slidesharecdn.com/2013-pythonpackaginganddependencyresolutionsolving1-150730194703-lva1-app6891/85/Python-packaging-and-dependency-resolution-10-320.jpg)
![use pipdeptree
$ pip freeze
A==1.0.0
B==1.0.0
C==1.0.0
$ pipdeptree
Warning!!! Possible confusing dependencies found:
* B==1.0.0 -> A [required: ==1.0.0, installed: 1.0.0]
C==1.0.0 -> A [required: >=2.0.0, installed: 1.0.0]
------------------------------------------------------------------------
wsgiref==0.1.2
B==1.0.0
- A [required: ==1.0.0, installed: 1.0.0]
C==1.0.0
- A [required: >=2.0.0, installed: 1.0.0]](https://image.slidesharecdn.com/2013-pythonpackaginganddependencyresolutionsolving1-150730194703-lva1-app6891/85/Python-packaging-and-dependency-resolution-37-320.jpg)
Python packaging and dependency resolution
- 1. Python Packaging and how to improve dependency resolution Tatiana Al-Chueyr Martins @tati_alchueyr PythonDay Pernambuco - 28th September 2013, Recife
- 2. tati_alchueyr.__doc__ ● computer engineer by UNICAMP ● senior software engineer at Globo.com ● open source enthusiastic ● pythonist since 2003 #opensource #python #android #arduino
- 3. Packaging overview Code Package Package Server
- 4. Code # life.py def cleanup_house(address): # .... def walk_dog(dog_name): # … class Man(object): __doc__ = “”
- 5. Package # world.py from life import Man
- 7. Packaging overview Code Package Package Server
- 9. Packaging creation pack # setup.py (...) $ python setup.py sdist
- 10. Packaging creation upload # ~/.pypirc [company] username: Andreia password: pwd repository: http://pypi.company.com $ python setup.py sdist upload -r company
- 12. Packaging usage search & download use $ easy_install life # something.py from life import Man
- 13. Package way to make the code available so developers can use it
- 14. Package setup.py - contains lots of metadata - dependencies - paths
- 15. Packages server: Cheese Shop place where developers can: ● find packages ● download packages ● upload packages
- 16. Brief on Python packaging history ● distutils ○ no dependecy management ○ problems between cross-platforms ○ no consistent way to reproduce an installation ○ not all metadata was handled ● setuptools: built on top of distutils ○ introduces easy_install ○ no way to uninstall installed packages ○ provides dependencies management ○ introduced eggs (similar to zip files) ● distribute: fork of setuptools ○ fork of setuptools ● distutils2 (discontinued?) ○ standard versioning (major.minor.micro) ○ setup.cfg: pulls metadata from setup.py file, without needing to run setup.py ○ which operating system requires which dependecy pysetup: their interations easyinstall and setuptools with disutils- extract stuff from setup. py
- 17. Distutils ● Started by Distutils SIG (Greg Ward) ● Added to stand lib in Python 1.6 (2000) ● solves ○ issues a variety of commands through setup.py (crete tarball, install your project, compiling C extensions of your python code) ● problems ○ no dependency management ○ problems between OS ○ no consistent way to reproduce an installation ○ not all metadata was handled
- 18. Brief on Python packaging history PEP 386: changing the version comparison modules PEP 376: database of installed python distributions PEP 345: metadata for python software packages 1.2
- 19. Chronology of Packaging by Ziade http://ziade.org/2012/11/17/chronology-of-packaging/
- 20. Chronology of Packaging by Ziade http://ziade.org/2012/11/17/chronology-of-packaging/
- 21. Brief on Python packaging history old busted new hawtness setuptools -> distribute easy_install -> pip system python -> virtual-env
- 22. Virtualenv “virtualenv is a tool to create isolated Python environments.” https://pypi.python.org/pypi/virtualenv
- 23. VirtualenvWrapper “virtualenvwrapper is a set of extensions to Ian Bicking's virtualenv tool” -- Doug Hellmann https://pypi.python.org/pypi/virtualenvwrapper $ mkvirtualenv <name> --python= --no-site-packages= --system-site-packages= $ rmvirtualenv $VIRTUALENVWRAPPER_HOOK_DIR/initialize
- 24. Pip A tool for installing and managing Python packages. http://www.pip-installer.org/en/latest/index.html $ pip search numpy $ pip help $ pip install flask $ pip uninstall django $ pip freeze --no-deps --extra-index-url --index-url --download-cache --proxy --no-install git / egg / ... pip install -r requirements.txt
- 25. Pip (...) “This allows users to be in control of specifying an environment of packages that are known to work together.” (...) http://www.pip-installer.org/en/latest/cookbook.html
- 26. How Pip deals with dependency inconsistencies?
- 27. Pip install -r requirements.txt B A # requirements.txt B C # B/setup.py A==1.0.0 # C/setup.py A>=2.0.0 C what version of A is installed? $ pip install -r requirements.txt
- 28. Pip install -r requirements.txt # requirements.txt B C # B/setup.py A==1.0.0 # C/setup.py A>=2.0.0 $ pip freeze A==1.0.0 B==1.0.0 C==1.00. B A C
- 29. Pip install -r requirements.txt # requirements.txt C B # B/setup.py A==1.0.0 # C/setup.py A>=2.0.0 what happens? error? $ pip install -r requirements.txt B A C
- 30. Pip install -r requirements.txt # requirements.txt C B # B/setup.py A==1.0.0 # C/setup.py A>=2.0.0 $ pip freeze A==2.0.0 B==1.0.0 C==1.00. B A C
- 31. Pip install -r requirements.txt # requirements.txt C B A==1.5.0 # B/setup.py A==1.0.0 # C/setup.py A>=2.0.0 what happens? error? $ pip install -r requirements.txt B A C
- 32. Pip install -r requirements.txt # requirements.txt C B A==1.5.0 # B/setup.py A==1.0.0 # C/setup.py A>=2.0.0 $ pip freeze A==1.5.0 B==1.0.0 C==1.00. B A C
- 33. Explanation Considering pip 1.5.4: ● pip doesn’t identify conflicts of interest between dependency packages ● why? ○ pip solves dependencies analyzing them in a list ○ it only concerns in solving the dependencies of the package being analyzed at that moment ○ the last package dependencies prevail
- 34. provided a package at pypi, how do I know its dependencies?
- 35. provided a package at pypi, how do I know its dependencies? manually looking to them
- 36. dependencies of a package if you install a package, you can use: $ pip show C To show dependencies, but they don’t contain versions - only packages names
- 37. use pipdeptree $ pip freeze A==1.0.0 B==1.0.0 C==1.0.0 $ pipdeptree Warning!!! Possible confusing dependencies found: * B==1.0.0 -> A [required: ==1.0.0, installed: 1.0.0] C==1.0.0 -> A [required: >=2.0.0, installed: 1.0.0] ------------------------------------------------------------------------ wsgiref==0.1.2 B==1.0.0 - A [required: ==1.0.0, installed: 1.0.0] C==1.0.0 - A [required: >=2.0.0, installed: 1.0.0]
- 38. Does the requirements.txt assure your environment will be reproduced always the same?
- 39. Does the requirements.txt assure your environment will be reproduced always the same? not necessarily
- 40. requirements.txt if you want to assert the same behavior in all installations: ● don’t use >=, <=, >, < ● pin all dependencies (even deps of deps) ● pin exactly (==)
- 41. some extra notes
- 42. Have your own pypi / proxy old versions might be removed from remote repositories the repository might be down during a deploy, and can crash your application
- 43. Have your own pypi / proxy
- 44. Have your own pypi / proxy host a PyPI mirror (bandersnatch, pep381client) host a PyPI cache (devp) PyPI server implementations: ● resilient (devpi) ● AWS S3 PyPI server (pypicloud) ● minimalistic PyPI (pypiserver) ● PyPI written in Django (chishop, djangopypi) Many others..! At globo.com we have both a PyPI server and a PyPI cache proxy.
- 45. dumb ways to manage your dependencies….
- 47. 1. understand the tools you use to manage dependencies 2. keep your dependencies up to date, but take care with >= / > 3. take care of your cheese-shop
- 49. thanks! slideshare: @alchueyr questions? Tatiana Al-Chueyr Martins @tati_alchueyr