pip 6.0 broke "pip install --pre --upgrade --find-links"

[rgommers]

TravisCI for numpy started failing after the pip 6.0 release: numpy/numpy#5386

The reason is that pip install --pre --upgrade --find-links dist numpy, where dist contains a wheel built from current master, starts to install numpy 1.9.1 from PyPi instead of that wheel. I have replicated the issue locally in a virtualenv:

$ pip install --pre --upgrade --find-links ../../numpy/dist numpy  # works with 1.5.6
$ pip uninstall numpy
$ pip install --upgrade pip   # installed pip 6.0
$ pip install --pre --upgrade --find-links ../../numpy/dist numpy  # starts downloading from PyPi

I can bisect the issue in pip if needed, but I'm hoping someone can tell me straight away what changed.

[dstufft]

Can you post the final pip install with a -vvv added?

[rgommers]

Sure:

$ pip install -vvv --pre --upgrade --find-links ../../numpy/dist numpy
Collecting numpy
  Getting page https://pypi.python.org/simple/numpy/
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/numpy/ HTTP/1.1" 200 2596
  URLs to search for versions for numpy:
  * https://pypi.python.org/simple/numpy/
  Skipping link ../../numpy/dist (from -f); not a file
  Getting page https://pypi.python.org/simple/numpy/
  Analyzing links from page https://pypi.python.org/simple/numpy/

Didn't know about -vvv, looks useful. The "skipping link" is telling. I don't understand why the message is not a file, according to the docs it can be a path.

[dstufft]

Does it work if you use file:///full/path/to/numpy/dist instead?

[rgommers]

No, same message.

And just for completeness:

$ ls ../../numpy/dist
numpy-1.10.0.dev_243ab56-cp27-none-linux_i686.whl

[dstufft]

Hmm. That is very strange. I can't think of what would have changed between then and now. Let me see if I can reproduce locally.

[dstufft]

Oh!

I did just notice something. What happens if you change numpy-1.10.0.dev_243ab56-cp27-none-linux_i686.whl to numpy-1.10.0.dev+243ab56-cp27-none-linux_i686.whl.

[rgommers]

No change when changing _ to +. I hope a plus is not recommended in a versioning scheme now?

Edit: I see your suggestion also had no none. I copied the filename exactly now, still no change

[dstufft]

A + has a specific meaning in version schemes now (PEP 440 local versions).

Trying to repro this now.

[dstufft]

Hmm. I get this:

Collecting numpy
  URLs to search for versions for numpy:
  Skipping link ./dist (from -f); not a file
  Skipping file:///Users/dstufft/.virtualenvs/tmp-f87184b6fa6f379/dist/numpy-1.10.0.dev_243ab56-cp27-none-linux_i686.whl because it is not compatible with this Python

Wonder if this might be something specific to Linux.

[rgommers]

OK clear. I'm signing off for tonight. Thanks for looking into this.

[argriffing]

Looks superficially similar to #1029.

[dstufft]

I was able to reproduce the error, but changing the dev-<stuff> to dev+<stuff> fixed it. I believe this error is caused because the version isn't a valid PEP 440 version when you include the githash inside the version like that.

[argriffing]

With the new pip release, I think the latest development versions of numpy and scipy are considered older than any released version, whereas with the earlier pip version I think this was not the case. Numpy and scipy do not have valid PEP 440 version names.

"""
Check the relative ordering of package versions.

"""
from __future__ import print_function, division

import sys

import numpy
import scipy

import pip
import pkg_resources
import setuptools

def main():
    print('python version:', sys.version)
    print('numpy version:', numpy.__version__)
    print('scipy version:', scipy.__version__)
    print('pip version:', pip.__version__)
    print('setuptools version:', setuptools.__version__)
    print()

    version_names = ('1.14.2', '1.16.2', '1.17.0.dev-deadbeef')

    # I think this is the setuptools order.
    print('version order provided by setuptools pkg_resources:')
    version_objects = [
            pkg_resources.parse_version(x) for x in version_names]
    print(sorted(version_objects))
    print()

    # This should be the pip order.
    print('version order provided by the "vendored" setuptools pkg_resources '
            'provided by pip:')
    try:
        version_objects = [
                pip._vendor.packaging.version.parse(x) for x in version_names]
    except AttributeError:
        version_objects = [
                pip._vendor.pkg_resources.parse_version(x) for x in version_names]
    print(sorted(version_objects))
    print()

main()

1.5.6 output:

python version: 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2]
numpy version: 1.10.0.dev-243ab56
scipy version: 0.16.0.dev-619698b
pip version: 1.5.6
setuptools version: 8.2.1

version order provided by setuptools pkg_resources:
[<LegacyVersion('1.17.0.dev-deadbeef')>, <Version('1.14.2')>, <Version('1.16.2')>]

version order provided by the "vendored" setuptools pkg_resources provided by pip:
[('00000001', '00000014', '00000002', '*final'), ('00000001', '00000016', '00000002', '*final'), ('00000001', '00000017', '*@', '*deadbeef', '*final')]

6.0 output:

python version: 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2]
numpy version: 1.10.0.dev-243ab56
scipy version: 0.16.0.dev-619698b
pip version: 6.0
setuptools version: 8.2.1

version order provided by setuptools pkg_resources:
[<LegacyVersion('1.17.0.dev-deadbeef')>, <Version('1.14.2')>, <Version('1.16.2')>]

version order provided by the "vendored" setuptools pkg_resources provided by pip:
[<LegacyVersion('1.17.0.dev-deadbeef')>, <Version('1.14.2')>, <Version('1.16.2')>]

[argriffing]

... and when I change the version names to

version_names = (
            '1.14.2',
            '1.16.2',
            '1.17.0.dev-deadbeef',
            '1.17.0.dev+deadbeef',
            )

I get the order [<LegacyVersion('1.17.0.dev-deadbeef')>, <Version('1.14.2')>, <Version('1.16.2')>, <Version('1.17.0.dev0+deadbeef')>]. So I'm almost certain this is the problem.