venv on Windows with symlinks is broken if invoked with -I

[gaborbernat]

BPO	42013
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba, @gaborbernat

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-10-12.10:55:28.257>
labels = ['3.10', '3.9', 'OS-windows']
title = 'venv on Windows with symlinks is broken if invoked with -I'
updated_at = <Date 2020-10-12.22:39:02.541>
user = 'https://github.com/gaborbernat'

bugs.python.org fields:

activity = <Date 2020-10-12.22:39:02.541>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2020-10-12.10:55:28.257>
creator = 'gaborjbernat'
dependencies = []
files = []
hgrepos = []
issue_num = 42013
keywords = []
message_count = 12.0
messages = ['378485', '378487', '378488', '378490', '378491', '378492', '378493', '378494', '378499', '378507', '378512', '378532']
nosy_count = 6.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'gaborjbernat']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue42013'
versions = ['Python 3.9', 'Python 3.10']

Linked PRs

• gh-86179: Implement realpath() on Windows during getpath.py calculations #113033
• gh-86179: Avoid making case-only changes when calculating realpath() during initialization #113077
• [3.12] gh-86179: Implement realpath() on Windows for getpath.py calculations (GH-113033) #113121
• gh-86179: Skip test case that fails on POSIX with unversioned binary #114136

[gaborbernat]

Here's a small reproducible, run it on a Windows OS that has symlinks enabled:

import shutil
import venv
import subprocess

shutil.rmtree("venv", ignore_errors=True)
venv.EnvBuilder(with_pip=False, symlinks=True).create("venv")

# works
subprocess.check_call(["venv\\Scripts\\python.exe", "-c", "import sys; print(sys.executable)"])

# fails with No module named 'encodings'
subprocess.check_call(["venv\\Scripts\\python.exe", "-Ic", "import sys; print(sys.executable)"])

[eryksun]

I can't reproduce the issue with the normal 3.9.0 distribution from python.org. For example:

    >>> venv.EnvBuilder(with_pip=False, symlinks=True).create("venv")
    >>> subprocess.check_call(["venv\\Scripts\\python.exe", "-Ic", "import sys; print(sys.executable)"])
    C:\Temp\env\venv\Scripts\python.exe
    0

Which Python version(s) and distribution(s) did you test? Do you have the PYTHONHOME and/or PYTHONPATH environment variables set?

[gaborbernat]

❯ py -c 'import sys; print(sys.version)'
3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)]

[FFY00]

This error most likely happens because sys.path isn't being set properly and the standard library isn't being included.

We need more information about the specific environment this is triggering.

Which CPython build are you using? The one from Github Actions, right?
We also need the environment variables.

[FFY00]

Well, actually the environment variables /should/ not matter as -I implies -E.

[gaborbernat]

Ok, the missing link is that the python you run into needs to be also a symlink venv:

❯ py -m venv env --without-pip --clear --symlinks
❯ .\env\Scripts\python.exe -c "import venv, subprocess; venv.EnvBuilder(with_pip=False, symlinks=True).create('venv'); subprocess.check_call(['venv\\Scripts\\python.exe', '-Ic', 'import sys; print(sys.executable)'])"

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

[gaborbernat]

We need more information about the specific environment this is triggering.

Both Gitub Actions Windows CPython3.9 or installer as downloaded from (on Windows 10) https://www.python.org/downloads/release/python-390/.

PYTHONHOME should not be set all. PYTHONPATH needs to be set carefully. It should never include standard-library path

The issue manifests independent of those environment variables, netiher of them are set.

[eryksun]

Well, actually the environment variables /should/ not matter as -I
implies -E.

The operative word there is "should". I was grasping for anything that might explain why I couldn't reproduce the issue.

Ok, the missing link is that the python you run into needs to
be also a symlink venv

Thank you, that reproduces the problem for me.

[eryksun]

This issue is partly due to bpo-8901, which changed the behavior of the -E and -I command-line options to make them ignore the default PythonPath value in the registry key "Software\Python\PythonCore\X.Y\PythonPath". The change itself is not wrong. It's just exposing an underlying problem.

The home path in pyvenv.cfg is from sys._base_executable. In a launcher-based environment that's created from the base installation, sys._base_executable is the real base executable. OTOH, in a symlink-based virtual environment, sys._base_executable is the same as sys.executable. Consequently, if a virtual environment is created from a symlink-based virtual environment, the home path in pyvenv.cfg refers to the creating environment instead of the base installation. In this case, with the current implementation, the standard library can only be found by falling back on the default PythonPath in the registry.