SciPy22 - Building binary extensions with pybind11, scikit build, and cibuildwheel
0 likes901 views
The document details advancements and features of pybind11, a C++11 interface for Python, highlighting its ease of integration, performance improvements, and new capabilities across various versions up to 2.10. It also discusses the upcoming features, collaborations, and tools like scikit-build and cibuildwheel to enhance Python package building with compiled extensions. Additionally, it addresses changes in Python support, documentation updates, and future plans for improving user experience and library interoperability.
![Maintenance: Release notes
9
Somewhat interesting release note system
Pull requests get a template to
fi
ll out
<!--
Title (above): please place [branch_name] at the beginning if you are targeting a branch other than master.
*Do not target stable*. It is recommended to use conventional commit format, see conventionalcommits.org, but
not required.
-->
## Description
<!-- Include relevant issues or PRs here, describe what changed and why -->
## Suggested changelog entry:
<!-- Fill in the below block with the expected RestructuredText entry. Delete if no entry needed;
but do not delete header or rst block if an entry is needed! Will be collected via a script. -->
```rst
```
<!-- If the upgrade guide needs updating, note that here too -->](https://image.slidesharecdn.com/scipy22-binaryextensions-220816153351-1b67bff5/85/SciPy22-Building-binary-extensions-with-pybind11-scikit-build-and-cibuildwheel-10-320.jpg)
![Maintenance: Release notes (2)
10
GHA bot tags PR’s after they get merged
name: Labeler
on:
pull_request_target:
types: [closed]
jobs:
label:
name: Labeler
runs-on: ubuntu-latest
steps:
- uses: actions/labeler@main
if: github.event.pull_request.merged == true
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
configuration-path: .github/labeler_merged.yml + some labeler con
fi
g
fi
les
You might need
permissions: contents: …](https://image.slidesharecdn.com/scipy22-binaryextensions-220816153351-1b67bff5/85/SciPy22-Building-binary-extensions-with-pybind11-scikit-build-and-cibuildwheel-11-320.jpg)
![Bonus package: build
14
SDist and wheel builder
pipx run build Builds SDist from source, wheel from SDist
pipx run build --sdist --wheel Builds SDist and wheel from source
New features this year Planned
Prettier printout
Better error messages
Python 3.11 support
Use Flit (better bootstrapping)
Redesign environment support
Threadsafe API
.github/work
fl
ows/wheels.yml
on: [push, pull_request]
jobs:
build_wheels:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build wheels
run: pipx run build
- uses: actions/upload-artifact@v3
with:
path: ./wheelhouse/*.whl](https://image.slidesharecdn.com/scipy22-binaryextensions-220816153351-1b67bff5/85/SciPy22-Building-binary-extensions-with-pybind11-scikit-build-and-cibuildwheel-15-320.jpg)
![Package: 🎡 cibuildwheel
15
Multiplatform redistributable wheel builder .github/work
fl
ows/wheels.yml
on: [push, pull_request]
jobs:
build_wheels:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os:
- ubuntu-22.04
- windows-2022
- macos-11
steps:
- uses: actions/checkout@v4
- name: Build wheels
uses: pypa/cibuildwheel@v2.8.0
- uses: actions/upload-artifact@v3
with:
path: ./wheelhouse/*.whl
Supports all major CI providers
GitHub Action provided too!
Can run locally, on servers, etc.
2.0 was released at last SciPy!
Supports:
Targeting macOS 10.9+
Apple Silicon cross-compiling 3.8+
All variants of manylinux (including emulation)
musllinux
PyPy 3.7-3.9
Repairing and testing wheels
Reproducible pinned defaults (can unpin)
A
ffi
liated (shared maintainer) with manylinux
Close collaboration with PyPy devs
Joined the PyPA in 2021
Used by matplotlib, mypy, scikit-learn, and more](https://image.slidesharecdn.com/scipy22-binaryextensions-220816153351-1b67bff5/85/SciPy22-Building-binary-extensions-with-pybind11-scikit-build-and-cibuildwheel-16-320.jpg)
![Pure Python Packaging
22
# pyproject.toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "example"
version = "0.1.0"
Many pure-Python backends today!
Hatch, PDM, Flit, and more!
Even modern setuptools supports this
But what if you want to add compiled extensions?
Fall back 8+ years into distutils/setuptools hacks
Not tied to a single solution!
Adding distutils to the stdlib was a disaster
Setuptools extending distutils was also a disaster
But modern standards free us from this!
See https://scikit-hep.org/developer
Or https://packaging.python.org](https://image.slidesharecdn.com/scipy22-binaryextensions-220816153351-1b67bff5/85/SciPy22-Building-binary-extensions-with-pybind11-scikit-build-and-cibuildwheel-24-320.jpg)
![Extensionlib (w/ PEP?)
26
Since the proposal, “extensionlib” and/or a PEP
has been discussed at PyCon 2022. A PoC lib by
@ofek exists. Stay tuned!
Provides a common API for build-
backends to extension creation tools
We could be a plugin for all builders via a single interface!
[build-system]
requires = ["hatchling", “scikit-build-core”, "mypyc"]
build-backend = "hatchling.build"
[project]
name = "example"
version = "0.1.0"
[[build-system.extensions]]
build-backend = "scikit_build_core.extension"
src = "src1/CMakeLists.txt"
[[build-system.extensions]]
build-backend = "scikit_build_core.extension"
src = "src2/CMakeLists.txt"
[[build-system.extensions]]
build-backend = "mypyc.extension"
src = "src/*.py"
(Rough example of idea)](https://image.slidesharecdn.com/scipy22-binaryextensions-220816153351-1b67bff5/85/SciPy22-Building-binary-extensions-with-pybind11-scikit-build-and-cibuildwheel-31-320.jpg)
![Modern (simple) packaging
35
pyproject.toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "package"
version = "0.1.2"
No setup.py, setup.cfg, or MANIFEST.in required!](https://image.slidesharecdn.com/scipy22-binaryextensions-220816153351-1b67bff5/85/SciPy22-Building-binary-extensions-with-pybind11-scikit-build-and-cibuildwheel-41-320.jpg)
SciPy22 - Building binary extensions with pybind11, scikit build, and cibuildwheel
- 1. Henry Schreiner Princeton University Joe Rickerby Nord Projects Ralf Grosse-Kunstleve Google Wenzel Jakob Ecole Polytechnique Fédérale de Lausanne Matthieu Darbois PyPA Aaron Gokaslan Facebook Research Jean-Christophe Fillion-Robin Kitware Matt McCormick Kitware Building binary extensions with pybind11, scikit-build, and cibuildwheel Maintainers track SciPy 2022 - Austin July 15, 2022
- 2. Package: pybind11 2 Header-only pure C++11 CPython/PyPy interface Trivial to add to a project No special build requirements No dependencies No precompile phase Not a new language Think of it like the missing C++ API for CPython Designed for one purpose!
- 3. Example of usage 3 #include <pybind11/pybind11.h> int add(int i, int j) { return i + j; } PYBIND11_MODULE(example, m) { m.def("add", &add); } Standard include Normal C++ to bind Create a Python module Signature statically inferred Docs and parameter names optional g++ -shared -fPIC example.cpp $(pipx run pybind11 --includes) -o example$(python3-config --extension-suffix) Complete, working, no-install example (linux version)!
- 4. Many great features 4 Simple enough for tiny projects 714K code mentions of pybind11 on GitHub Powerful enough for huge projects SciPy, PyTorch, dozens of Google projects Small binaries prioritized Perfect for WebAssembly with Pyodide Powerful object lifetime controls py::keep_alive, std::shared_ptr, and more NumPy support without NumPy headers No need to lock minimum NumPy at build Supports interop in both directions Can even be used to embed Python in C++ Most STL containers and features supported Including C++17 additions, like std::variant (or boost) Vectorize methods or functions py::vectorize, even on a lambda function Trampoline classes and multiple inheritance Complex C++ is supported Complete control of the Python representation Special methods, inheritance, pickling, and more Bu ff er protocol and array classes Includes Eigen support too Cross-extension ABI One extension can return a type wrapped in another one
- 5. New in 2.8 5 2.7 was released at last SciPy! py::raise_from Chaining exceptions Local exception mapping Custom rules for each module Array improvements View and reshape arrays Custom __new__ support Fixing old bug Python builtins as callbacks Better consistency Embedded interpreter improvements argv setting, __ fi le__ set py::slice None support Using std::optional Rerun CMake with di ff erent Python Updates cache if stale mingw support added Tested in CI now too py::custom_type_setup Advanced GC tricks and more
- 6. New in 2.9 6 py::args can be followed No longer required to be last py::const_name replaces py::_ More readable, gettext clash removed More built-in exception chaining Using recent py::raise_from addition py::multiple_inheritance detection Only explicitly required if bases are hidden Codebase formatted with clang-format Using clang-format wheel & pre-commit Better support for std::string_view Interoperable with built-in types Lots of new checks, small fi xes, better support for PyPy, CPython 3.11, etc.
- 7. New in 2.10 7 Removed Python 2.7 & 3.5 support Removed 1K+ LoC, simpler, cleaner Removed MSVC 2015 (& limited 2017) Hard to fi nd & not very useful with 3.5 gone New docs theme (Furo) Cleaner, ToC for pages, dark-mode Support std::monostate std::variant as optional or non-constructible Better exception handling Easier without Python 2.7 py::any set and py::frozenset Copy to set (like py::set) py::capsule::set_name Can manipulate later (DLPack) NumPy type enhancements More accessors and type number constructor Python 3.11 beta support First version working with all changes CMake fi xes Needed for Pyodide (web assembly) & VCPKG
- 8. New in 2.10 7 Removed Python 2.7 & 3.5 support Removed 1K+ LoC, simpler, cleaner Removed MSVC 2015 (& limited 2017) Hard to fi nd & not very useful with 3.5 gone New docs theme (Furo) Cleaner, ToC for pages, dark-mode Support std::monostate std::variant as optional or non-constructible Better exception handling Easier without Python 2.7 py::any set and py::frozenset Copy to set (like py::set) py::capsule::set_name Can manipulate later (DLPack) NumPy type enhancements More accessors and type number constructor Python 3.11 beta support First version working with all changes CMake fi xes Needed for Pyodide (web assembly) & VCPKG 2.10.0 releasing today
- 9. Upcoming plans 8 Refactors Decoupling unit tests Breaking into smaller, IWUY headers https://github.com/pybind/pybind11/wiki/Roadmap Optional Precompilation Would dramatically speed up compile Huge change, so being held back by other work Merge the smart holder branch Full interoperability with std::unique_ptr and std::shared_ptr Opt-in with <pybind11/smart_holder.h> Safe passing to C++, including trampolines Further ideas (not promised yet) Better MyPy integration (some minor work done) Better Scikit-build integration Upstreaming some improvements from nanobind (next slide)
- 10. Maintenance: Release notes 9 Somewhat interesting release note system Pull requests get a template to fi ll out <!-- Title (above): please place [branch_name] at the beginning if you are targeting a branch other than master. *Do not target stable*. It is recommended to use conventional commit format, see conventionalcommits.org, but not required. --> ## Description <!-- Include relevant issues or PRs here, describe what changed and why --> ## Suggested changelog entry: <!-- Fill in the below block with the expected RestructuredText entry. Delete if no entry needed; but do not delete header or rst block if an entry is needed! Will be collected via a script. --> ```rst ``` <!-- If the upgrade guide needs updating, note that here too -->
- 11. Maintenance: Release notes (2) 10 GHA bot tags PR’s after they get merged name: Labeler on: pull_request_target: types: [closed] jobs: label: name: Labeler runs-on: ubuntu-latest steps: - uses: actions/labeler@main if: github.event.pull_request.merged == true with: repo-token: ${{ secrets.GITHUB_TOKEN }} configuration-path: .github/labeler_merged.yml + some labeler con fi g fi les You might need permissions: contents: …
- 12. Maintenance: Release notes (3) 11 Script runner in nox @nox.session(reuse_venv=True) def make_changelog(session: nox.Session) -> None: """ Inspect the closed issues and make entries for a changelog. """ session.install("ghapi", "rich") session.run("python", "tools/make_changelog.py") Controls script dependencies
- 13. PRs can be updated, fi nal polish on copy/paste Labels manually removed (in bulk)
- 14. Bonus package: nanobind 13 C++17+ & Python 3.8+ only Similar API to pybind11 Intentionally more limited than pybind11 Focus on small, e ffi cient bindings Some ideas can be backported to pybind11 https://github.com/wjakob/nanobind
- 15. Bonus package: build 14 SDist and wheel builder pipx run build Builds SDist from source, wheel from SDist pipx run build --sdist --wheel Builds SDist and wheel from source New features this year Planned Prettier printout Better error messages Python 3.11 support Use Flit (better bootstrapping) Redesign environment support Threadsafe API .github/work fl ows/wheels.yml on: [push, pull_request] jobs: build_wheels: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Build wheels run: pipx run build - uses: actions/upload-artifact@v3 with: path: ./wheelhouse/*.whl
- 16. Package: 🎡 cibuildwheel 15 Multiplatform redistributable wheel builder .github/work fl ows/wheels.yml on: [push, pull_request] jobs: build_wheels: runs-on: ${{ matrix.os }} strategy: matrix: os: - ubuntu-22.04 - windows-2022 - macos-11 steps: - uses: actions/checkout@v4 - name: Build wheels uses: pypa/[email protected] - uses: actions/upload-artifact@v3 with: path: ./wheelhouse/*.whl Supports all major CI providers GitHub Action provided too! Can run locally, on servers, etc. 2.0 was released at last SciPy! Supports: Targeting macOS 10.9+ Apple Silicon cross-compiling 3.8+ All variants of manylinux (including emulation) musllinux PyPy 3.7-3.9 Repairing and testing wheels Reproducible pinned defaults (can unpin) A ffi liated (shared maintainer) with manylinux Close collaboration with PyPy devs Joined the PyPA in 2021 Used by matplotlib, mypy, scikit-learn, and more
- 17. Manylinux 16 manylinux1 manylinux2010 manylinux_2_24 manylinux2014 manylinux_2_28 RHEL 5 (CentOS) RHEL 6 (CentOS) RHEL 7 (CentOS) Debian 9 RHEL 8 (Almalinux) Support ended Jan 1, 2022 Updated till CI breaks Support ending Aug 1, 2022 Support likely to end Default in cibuildwheel musllinux_1_1 Alpine 3.12 MUSL distributions
- 18. Cibuildwheel: what’s new 17 New in cibuildwheel 2.1-2.4 Local Windows & MacOS runs TOML overrides array manylinux2014 default musllinux Environment variable passthrough Experimental Windows ARM New in cibuildwheel 2.5-2.8 ABI3 wheel support Build from SDist tomllib on Python 3.11 (host) CPython 3.11 beta support manylinux_2_28 support (RHEL 8) Linux builds from Windows Podman support Support for Pythonless wheels In Development New setup-python environment Python 3.6 removed as host Python 3.11b4 in progress as of today
- 19. Cibuildwheel: what might be 18 Cross compiling Windows ARM Supporting setuptools, hopefully more Cross compiling Linux Stuck with missing RH dev toolkit trick Local run improvements Skip an environment variable
- 20. cibuildwheel tips 19 Build wheels locally pipx run cibuildwheel --platform linux i i i i i i : d d d d d d d d d d d d d d d : " " " " " " " " " " " u u u u u u u u u u u u : Keep the GitHub Action up-to-date! .github/dependabot.yml version: 2 updates: - package-ecosystem: "github-actions" directory: "/" schedule: interval: "daily" Use pyproject.toml con fi g
- 21. cibuildwheel tips 19 Build wheels locally pipx run cibuildwheel --platform linux Keep the GitHub Action up-to-date! .github/dependabot.yml version: 2 updates: - package-ecosystem: "github-actions" directory: "/" schedule: interval: "daily" No longer needed, Dependabot respects version level! (v1 -> v2, not v2.0.0) Use pyproject.toml con fi g
- 22. GHA: building a composite action 20 GitHub Action’s new [email protected] feature is perfect for composite pipx actions! name: mypkg description: 'Runs a package' runs: using: composite steps: - uses: actions/setup-python@v4 id: python with: python-version: "3.7 - 3.10" update-environment: false - run: > pipx run --python '${{ steps.python.outputs.python-path }}' --spec '${{ github.action_path }}' mypkg shell: bash No side e ff ects! Used in nox & cibuildwheel Package you want to run
- 23. Package: scikit-build 21 CMake based build backend for Python Scikit-build is a CMake-setuptools adaptor from KitWare, the makers of CMake First introduced as PyCMake at SciPy 2014 and renamed in 2016 Includes CMake for Python and ninja for Python CMake and Ninja for Python All manylinux archs • Apple Silicon • cibuildwheel • nox Currently designed as a setuptools wrapper But that is changing…
- 24. Pure Python Packaging 22 # pyproject.toml [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [project] name = "example" version = "0.1.0" Many pure-Python backends today! Hatch, PDM, Flit, and more! Even modern setuptools supports this But what if you want to add compiled extensions? Fall back 8+ years into distutils/setuptools hacks Not tied to a single solution! Adding distutils to the stdlib was a disaster Setuptools extending distutils was also a disaster But modern standards free us from this! See https://scikit-hep.org/developer Or https://packaging.python.org
- 25. Packaging timeline 23 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 Scikit-build introduced (as PyCMake) PEP 517/518 enables non-setuptools backends to be developed and used PEP 621 de fi nes a standard con fi g format PEP 632 removes distutils from the standard library in Python 3.12 Proposed project PEP 660 editable installs Python Enhancement Proposals (PEPs) drive Python development - and provide standards for packaging now too! Before 2015, the only method to distribute packages was distutils/setuptools Good pure Python backends exist now (hatchling) - scikit-build could be a great compiled / CMake one!
- 26. Scikit-build (current design) 24 setup.py setup(…) Extensions Command dict Config settings distutils setuptools MANIFEST.in setup.cfg pyproject.toml CMakeLists.txt Raw setuptools for C++? No C++ standard support No parallel compiles No common libraries No custom libraries Hacks for cross compiling NumPy alone needs 13K lines to support this!
- 27. Scikit-build (current design) 24 setup.py setup(…) Extensions Command dict Config settings distutils setuptools MANIFEST.in setup.cfg pyproject.toml CMakeLists.txt Raw setuptools for C++? No C++ standard support No parallel compiles No common libraries No custom libraries Hacks for cross compiling NumPy alone needs 13K lines to support this! scikit-build Scikit-build wraps CMake! Best in class feature support ~60% of all C++ projects Very popular in the sciences (incl. HEP) Common and custom library support
- 28. Scikit-build (current design) 24 setup.py setup(…) Extensions Command dict Config settings distutils setuptools Problems with setuptools bridge Inherits problems from setuptools Deep dependencies on internals Poor interactions with new standards Deep nesting is hard to work with / debug Internals poorly documented Not composable with other plugins MANIFEST.in setup.cfg pyproject.toml CMakeLists.txt Raw setuptools for C++? No C++ standard support No parallel compiles No common libraries No custom libraries Hacks for cross compiling NumPy alone needs 13K lines to support this! scikit-build Scikit-build wraps CMake! Best in class feature support ~60% of all C++ projects Very popular in the sciences (incl. HEP) Common and custom library support
- 29. Scikit-build (new design) 25 pyproject.toml CMakeLists.txt build-system require scikit-build-core project configuration Name, etc. tool.scikit-build Optional custom config Planned other interfaces: Extension-based setuptools wrapper Can be used to support classic “scikit-build” Plugin mode (see next page) Design considerations: Good testing suite Excellent error messages Public API CMake code: Redesign with modern “FindPython” support Provide extension discovery system You could use pybind11 from pip in CMake! (Scikit-build-core may be a placeholder name)
- 30. Extensionlib (w/ PEP?) 26 Since the proposal, “extensionlib” and/or a PEP has been discussed at PyCon 2022. A PoC lib by @ofek exists. Stay tuned! Provides a common API for build- backends to extension creation tools We could be a plugin for all builders via a single interface!
- 31. Extensionlib (w/ PEP?) 26 Since the proposal, “extensionlib” and/or a PEP has been discussed at PyCon 2022. A PoC lib by @ofek exists. Stay tuned! Provides a common API for build- backends to extension creation tools We could be a plugin for all builders via a single interface! [build-system] requires = ["hatchling", “scikit-build-core”, "mypyc"] build-backend = "hatchling.build" [project] name = "example" version = "0.1.0" [[build-system.extensions]] build-backend = "scikit_build_core.extension" src = "src1/CMakeLists.txt" [[build-system.extensions]] build-backend = "scikit_build_core.extension" src = "src2/CMakeLists.txt" [[build-system.extensions]] build-backend = "mypyc.extension" src = "src/*.py" (Rough example of idea)
- 32. NSF Proposal 27 Scikit-build family: scikit-build, cmake, ninja, moderncmakedomain, scikit-build-examples pybind family: pybind11, python_example, cmake_example, scikit_build_example PyPA family: cibuildwheel, build scikit-hep family: Developer pages, cookie Updates can touch many repos! Year 1 plans: Introduce scikit-build-core: modern PEP 517 backend Support PEP 621 con fi guration Support use as plugin (possibly via extesionlib) Tighter CMake integration (con fi g from PyPI packages) Distutils-free code ready for Python 3.12 Year 2 plans: Convert projects to Scikit-build Year 3 plans: Website, tutorials, outreach https://iscinumpy.dev/post/scikit-build-proposal/
- 33. NSF Proposal 27 Scikit-build family: scikit-build, cmake, ninja, moderncmakedomain, scikit-build-examples pybind family: pybind11, python_example, cmake_example, scikit_build_example PyPA family: cibuildwheel, build scikit-hep family: Developer pages, cookie Updates can touch many repos! Year 1 plans: Introduce scikit-build-core: modern PEP 517 backend Support PEP 621 con fi guration Support use as plugin (possibly via extesionlib) Tighter CMake integration (con fi g from PyPI packages) Distutils-free code ready for Python 3.12 Year 2 plans: Convert projects to Scikit-build Year 3 plans: Website, tutorials, outreach On Wednesday, NSF 2209877 was awarded! https://iscinumpy.dev/post/scikit-build-proposal/
- 34. Scikit-build, since last time 28 CMake Current release (3.23) will likely drop manylinux1 wheels Scikit-build 0.13 ARM on Windows support MSVC 2022 support Target link libraries support via keyword Scikit-build 0.14 Install target Scikit-build 0.15 Cygwin support (Limited) FindPython support Final Python 2.7 & 3.5 release Final MSVC 2015 release + lots of fi xes & minor improvements
- 35. Questions? 29 Special thank you to Google for sponsoring our expanded CI, and for Ralf Grosse-Kunstleve’s time. Henry Schreiner supported by IRIS-HEP and the NSF under Cooperative Agreement OAC-1836650 To get in touch, currently we are using GitHub Discussions on all three packages.
- 36. henryiii henryschreiner3 Plumbum • POVM • PyTest GHA annotate-failures https://iscinumpy.dev https://scikit-hep.org https://iris-hep.org C++ & Python Building Python Packages Scikit-HEP: Other Other C++ Scikit-HEP: Histograms pybind11 (python_example, cmake_example, scikit_build_example) • Conda-Forge ROOT cibuildwheel • build • scikit-build (cmake, ninja, sample-projects) • Scikit-HEP/cookie boost-histogram • Hist • UHI • uproot-browser Vector • Particle • DecayLanguage • repo-review Other Python Jekyll-Indico Other Ruby CLI11 • GooFit Modern CMake • CMake Workshop Computational Physics Class Python CPU, GPU, Compiled minicourses Level Up Your Python My books and workshops
- 37. Scikit-HEP Developer Pages 31 Bonus slides (from Scikit-HEP talk yesterday)
- 38. Scikit-HEP developer pages 32 Universally useful advice for maintaining packages! Scikit-HEP was growing quickly How to keep quality high across all packages? scikit-hep.org/developer! Utilities Cookie-cutter Repo-review Topics Packaging (classic & simple) Style guide (pre-commit) Development setup pytest Static typing (MyPy) GitHub Actions (3 pages) Nox Maintained by nox & GitHub Actions Guided our other packages, like Awkward
- 40. Style 34 Pre-commit hooks Black Check-Manifest (setuptools only) Type checking PyCln Flake8 YesQA isort PyUpgrade Setup.cfg format (setuptools only) Spelling PyGrep hooks Prettier Clang-format (C++ only) Shellcheck (shell scripts only) PyLint (noisy)
- 41. Modern (simple) packaging 35 pyproject.toml [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [project] name = "package" version = "0.1.2" No setup.py, setup.cfg, or MANIFEST.in required!
- 42. Cookiecutter 36 pipx run cookiecutter gh:scikit-hep/cookie 11 backends to pick from Generation tested by nox In sync with the developer pages Setuptools Setuptools PEP 621 Flit Hatch PDM Poetry Scikit-build Setuptools C++ Maturin (Rust) + more!
- 43. Runs in WebAssembly locally! (via Pyodide)