Ship wheel packages for Linux ?

[atodorov]

What went wrong?

How do maintainers feel about providing wheel packages for Linux? Maybe 64bit only, same Python versions for which you build the Windows wheel packages ?

[aiudirog]

Building from source is significantly easier on Linux than on Windows, which is why I implemented the wheel deployments while adding Windows support. If we are going to do it, I would tie this into #192 instead of implementing it in Travis.

[frozencemetery]

I agree with @aiudirog - I'd rather not deal with writing more Travis goo.

In the interest of completeness, @atodorov, could you tell us more about your use case - specifically, is there anything that complicates either using the package manager's version or having pip build it?

[atodorov]

I am building docker images where we pip install gssapi. I don't want to have gcc or krb5-devel inside of them, just krb5-libs and the wheel package if possible.

[atodorov]

Another use case from me: building documentation on ReadTheDocs fails b/c we can't compile gssapi there

[frozencemetery]

Thanks, that's all good to know. I'm not opposed to doing this, but unfortunately it may take a while for me to get to it. Patches welcome :)

[atodorov]

@frozencemetery I will give it a try if somebody else doesn't beat me to it.

[tiran]

manylinux wheels are restricted to an ancient ABI (CentOS 5 or CentOS 6) and very few shared libraries. Unless you want to statically link in KRB5 and GSSAPI, it is very unlikely that you can get manylinux to work for python-gssapi

[frozencemetery]

The end result of what's in #205 now is:

HTTPError: 400 Client Error: Binary wheel 'gssapi-0.0.0.post18-cp27-cp27mu-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'. for url: https://upload.pypi.org/legacy/

As @tiran explained above, this is because Python prefers manylinux wheels. Since we can't staticly link in krb5 because it won't pass auditwheel, we need to be using manylinux2014 wheels. (Cred store extensions are added in 1.11, so they're not in el<=6, and I don't want to provide a version of python-gssapi without them.)

[tiran]

https://www.python.org/dev/peps/pep-0599/#the-manylinux2014-policy
https://pypi.org/project/auditwheel/

Build a wheel on CentOS 7 manylinux builder

$ docker run --network=host -i -t -v `pwd`:/io:Z quay.io/pypa/manylinux2014_x86_64 /bin/bash
# cd /io
# /opt/python/cp38-cp38/bin/pip install cython
# yum install -y krb5-devel
# /opt/python/cp38-cp38/bin/python3.8 setup.py bdist_wheel

Inspect the wheel

# auditwheel show dist/gssapi-1.6.5-cp38-cp38-linux_x86_64.whl 

gssapi-1.6.5-cp38-cp38-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libc.so.6 with versions {'GLIBC_2.2.5',
'GLIBC_2.16', 'GLIBC_2.3', 'GLIBC_2.17', 'GLIBC_2.14', 'GLIBC_2.3.4',
'GLIBC_2.7', 'GLIBC_2.8', 'GLIBC_2.4'}, libgssapi_krb5.so.2 with
versions {'gssapi_krb5_2_MIT'}, libdl.so.2 with versions
{'GLIBC_2.2.5'}, libpthread.so.0 with versions {'GLIBC_2.2.5'},
libkrb5support.so.0 with versions {'krb5support_0_MIT'},
libk5crypto.so.3 with versions {'k5crypto_3_MIT'}, libkrb5.so.3 with
versions {'krb5_3_MIT'}, libresolv.so.2 with versions {'GLIBC_2.2.5'},
libkeyutils.so.1 with versions {'KEYUTILS_0.3', 'KEYUTILS_1.5',
'KEYUTILS_1.0'}

This constrains the platform tag to "manylinux2014_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

Repair and try again

# auditwheel repair dist/gssapi-1.6.5-cp38-cp38-linux_x86_64.whl 
INFO:auditwheel.main_repair:Repairing gssapi-1.6.5-cp38-cp38-linux_x86_64.whl
INFO:auditwheel.wheeltools:Previous filename tags: linux_x86_64
INFO:auditwheel.wheeltools:New filename tags: manylinux2014_x86_64
INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp38-cp38-linux_x86_64
INFO:auditwheel.wheeltools:New WHEEL info tags: cp38-cp38-manylinux2014_x86_64
INFO:auditwheel.main_repair:
Fixed-up wheel written to /io/wheelhouse/gssapi-1.6.5-cp38-cp38-manylinux2014_x86_64.whl
# auditwheel show /io/wheelhouse/gssapi-1.6.5-cp38-cp38-manylinux2014_x86_64.whl 

gssapi-1.6.5-cp38-cp38-manylinux2014_x86_64.whl is consistent with the
following platform tag: "manylinux2014_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libpthread.so.0 with versions
{'GLIBC_2.2.5'}, libc.so.6 with versions {'GLIBC_2.2.5', 'GLIBC_2.14',
'GLIBC_2.3', 'GLIBC_2.3.4', 'GLIBC_2.4', 'GLIBC_2.8', 'GLIBC_2.7',
'GLIBC_2.16', 'GLIBC_2.17'}, libkrb5support-ec2f3e99.so.0.1 with
versions {'krb5support_0_MIT'}, libk5crypto-d17c94d0.so.3.1 with
versions {'k5crypto_3_MIT'}, libkrb5-0926b880.so.3.3 with versions
{'krb5_3_MIT'}, libdl.so.2 with versions {'GLIBC_2.2.5'},
libresolv.so.2 with versions {'GLIBC_2.2.5'}, libkeyutils-
dfe70bd6.so.1.5 with versions {'KEYUTILS_1.5', 'KEYUTILS_0.3',
'KEYUTILS_1.0'}, libgssapi_krb5-0f360511.so.2.2 with versions
{'gssapi_krb5_2_MIT'}

This constrains the platform tag to "manylinux2014_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

The resulting wheel works on my Fedora 32 box. But technically it is not a correct wheel. Please do not upload any binary wheels to PyPI until we have done thorough testing. I strongly recommend to upload any binary wheels to https://test.pypi.org/ for now and test them there.

[tiran]

Cython does not support PEP 384 limited API yet. There is some ongoing work to support it in 3.0.0 dev (CYTHON_LIMITED_API, Py_LIMITED_API). This means that you'd have to build a binary wheel for every Python version and CPU arch.

[tiran]

The binary wheel includes libgssapi_krb5 and changes the rpath of gssapi extensions to load its copy of libgssapi_krb5. When I load python-ldap I got two separate copies of libgssapi_krb5 loaded. Is this going to break features like MEMORY ccache?

$ python3 -m venv /tmp/venv
$ /tmp/venv/bin/pip install python-ldap six decorator
$ /tmp/venv/bin/pip install --index-url https://test.pypi.org/simple/ gssapi
Looking in indexes: https://test.pypi.org/simple/
Collecting gssapi
  Downloading https://test-files.pythonhosted.org/packages/55/3c/05d937a8131b9a12b97af440ac351d0709c7fc9c37e1684baff24f08d660/gssapi-1.6.5-cp38-cp38-manylinux2014_x86_64.whl (4.1MB)
...

$ /tmp/venv/bin/python
>>> import gssapi
>>> import ldap
>>> import os
>>> os.getpid()
995927

$ grep libgssapi_krb5 /proc/995927/maps 
7f907a2aa000-7f907a2b6000 r--p 00000000 fd:01 1850041                    /usr/lib64/libgssapi_krb5.so.2.2
7f907a2b6000-7f907a2f0000 r-xp 0000c000 fd:01 1850041                    /usr/lib64/libgssapi_krb5.so.2.2
7f907a2f0000-7f907a2fd000 r--p 00046000 fd:01 1850041                    /usr/lib64/libgssapi_krb5.so.2.2
7f907a2fd000-7f907a2fe000 ---p 00053000 fd:01 1850041                    /usr/lib64/libgssapi_krb5.so.2.2
7f907a2fe000-7f907a300000 r--p 00053000 fd:01 1850041                    /usr/lib64/libgssapi_krb5.so.2.2
7f907a300000-7f907a301000 rw-p 00055000 fd:01 1850041                    /usr/lib64/libgssapi_krb5.so.2.2
7f907babb000-7f907bb05000 r-xp 00000000 00:30 28592067                   /tmp/venv/lib/python3.8/site-packages/gssapi.libs/libgssapi_krb5-0f360511.so.2.2
7f907bb05000-7f907bd05000 ---p 0004a000 00:30 28592067                   /tmp/venv/lib/python3.8/site-packages/gssapi.libs/libgssapi_krb5-0f360511.so.2.2
7f907bd05000-7f907bd06000 r--p 0004a000 00:30 28592067                   /tmp/venv/lib/python3.8/site-packages/gssapi.libs/libgssapi_krb5-0f360511.so.2.2
7f907bd06000-7f907bd08000 rw-p 0004b000 00:30 28592067                   /tmp/venv/lib/python3.8/site-packages/gssapi.libs/libgssapi_krb5-0f360511.so.2.2
7f907bd08000-7f907bd1a000 rw-p 0004f000 00:30 28592067                   /tmp/venv/lib/python3.8/site-packages/gssapi.libs/libgssapi_krb5-0f360511.so.2.2

[simo5]

@tiran depending on linker options it may work better or worse, isn't there a way in wheels to tell to try to load the system library first and fall back to the embedded one only if a system one is not found ?
I guess this would cause issues if the system one is incompatible for some reason though ...

[simo5]

As for MEMORY cache, it uses thread local storage, so it might create silos between parts of the code using the two different libraries

[frozencemetery]

Right, the data structure(s) holding the MEMORY ccaches aren't exported, so I think it will just silo.

@tiran, it looks like the system copy is dragged in by ldap, right? So unless there's a way to ask for the system copy preferentially, I think we're going to end up with both...

[frozencemetery]

@atodorov The feeling I'm getting is that we're somewhat concerned about what happens when we try to provide manylinux wheels. I'm not sure I want to be debugging some of the potential problems we haven't ruled out, and the known issue around MEMORY ccaches is probably not good either.

One thing we could do is ship normal Linux wheels (i.e., not manylinux) from our github page as release artifacts. So they can't be on pypi, but you could still install them manually and not need C compiler etc.. Is this something that would be helpful to you?

@tiran, same question, but for IPA's use case? Also, any chance of a centos8-based manylinux having krb5 libraries in the "okay to link" set?

[atodorov]

One thing we could do is ship normal Linux wheels (i.e., not manylinux) from our github page as release artifacts. So they can't be on pypi, but you could still install them manually and not need C compiler etc.. Is this something that would be helpful to you?

Which Python version and OS version are these going to be built with? I have the feeling that unless consumers use the same this will be prone to errors.

Also I know you can manually install packages from other sources than PyPI but IDK how to specify the dependencies in such a way that they are transparent to the user.

Sounds like it won't work very well for my intended purposes.

[atodorov]

Another note:

IDK what manylinux packages are, nor how they are built but isn't this issue similar on Windows ? E.g. why is that on Windows we can ship a binary wheel and more or less expect it to work across various OS (Python?) versions but not on Linux ?

[tiran]

The situation on Windows is different. There is just one distribution (Microsoft Windows) with several versions and a strict backwards compatibility policy. DLL resolver works differently than on Linux, too.

There are tons of Linux distributions and each distribution has multiple versions with or without backwards compatibility. PyPI does not permit uploads of distribution specific packages. Only manylinux packages are allowed. Linux installations may or may not have libkrb5.so and similar, but manylinux standard does not permit to depend the libraries. A python-gssapi binary wheel has to ship old copies of these shared libraries. A process may load multiple incompatible copies of the shared libraries.

[jborean93]

Based on the conversation above this is not really possible to do for Linux, at least for the releases on PyPI. I personally don't think it's feasible to host our own wheels due to complexities of trying to make a "universal" wheel for Linux. Unless this landscape changes I don't think anything else can be done for Linux.

Thanks to everyone for investigating this feature and picking out the gritty parts here.

[gaborbernat]

Shouldn't https://github.com/pypa/cibuildwheel solve this problem now with a low effort?

[jborean93]

No the issue isn’t building the wheel it’s the fact that on Linux it needs to essentially ship it’s own libgssapi library from either MIT or Heimdal. This will be different from the system provided library that most other tools are linked to causing problems when a process tries to link multiple copies of the same library (at different versions). This is a requirement of manylinux wheels uploaded to PyPI and we can’t just have an external dep on libgssapi/libkrb5 because:

• it’s against the policy and not allowed, except for things like libc all other external need to be shipped with the wheel
• even if we could have an external dep there are no guarantees that the lib is accessible on the host - especially when Heimdal is used causing issues at runtime

[tiran]

libkrb5 / libgssapi use process-wide global states. For example one part of our application acquires credentials and stores it in global state. Another part of the application can use the credentials to log into a database. My comment #200 (comment) shows an example with a binary wheel of python-gssapi and a system build of python-ldap. As you can see there are two copies of libkrb5 in the process space, the bundled copy from python-gssapi and /usr/lib64/libgssapi_krb5.so.2.2, which is loaded by libldap.so. Although the are the same version and ABI, each copy has its own global state.

[gaborbernat]

Thanks for the detailed explanation 👍 Makes sense now.