pathlib: "Incorrect function" during resolve()

[earonesty]

BPO	31842
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba, @earonesty, @earonesty, @zombie110year, @christopher.pickering

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-10-22.21:31:13.027>
labels = ['type-bug', '3.9', 'OS-windows']
title = 'pathlib: "Incorrect function" during resolve()'
updated_at = <Date 2021-11-13.12:33:39.598>
user = 'https://github.com/earonesty'

bugs.python.org fields:

activity = <Date 2021-11-13.12:33:39.598>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2017-10-22.21:31:13.027>
creator = 'earonesty2'
dependencies = []
files = []
hgrepos = []
issue_num = 31842
keywords = []
message_count = 8.0
messages = ['304767', '356199', '357702', '357725', '381676', '390089', '406212', '406274']
nosy_count = 11.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'earonesty', 'earonesty2', 'zombie110year', 'maciozo', 'christopherpickering', 'riffitos']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31842'
versions = ['Python 3.9']

Linked PRs

• gh-76023: WinFSP giving WinError 1005 fix #128316
• gh-76023: WinFSP giving WinError 1005 fix (Python 3.14.0a3) #128328
• [3.14] gh-76023: Make os.path.realpath to ignore WinError 1005 in non-strict mode (GH-128328) #134203
• [3.13] gh-76023: Make os.path.realpath to ignore WinError 1005 in non-strict mode (GH-128328) #134204

[earonesty]

When strict is "false", pathlib should not fail if the network share is inaccessible. It should, as documented, catch the exception and continue with as much of the path as it has.

>>> pathlib.Path("v:").resolve()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\erik\AppData\Local\Programs\Python\Python36\lib\pathlib.py", line 1119, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "C:\Users\erik\AppData\Local\Programs\Python\Python36\lib\pathlib.py", line 193, in resolve
    s = self._ext_to_normal(_getfinalpathname(s))
OSError: [WinError 1] Incorrect function: 'v:'

[maciozo]

Same error occurs when attempting to resolve a path on an ImDisk RAM disk:

>>> pathlib.Path("T:\\").resolve()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Users\maciozo\AppData\Local\Continuum\miniconda3\envs\brainboxes\lib\pathlib.py", line 1151, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "C:\Users\maciozo\AppData\Local\Continuum\miniconda3\envs\brainboxes\lib\pathlib.py", line 202, in resolve
    s = self._ext_to_normal(_getfinalpathname(s))
OSError: [WinError 1] Incorrect function: 'T:\\'

[Zombie110year]

Same Error.

It happend by resolving a path which is not under C: driver,
in Windows System.

>>> Path().resolve()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\zom\scoop\apps\miniconda3\current\envs\web\lib\pathlib.py", line 1159, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "C:\Users\zom\scoop\apps\miniconda3\current\envs\web\lib\pathlib.py", line 202, in resolve
    s = self._ext_to_normal(_getfinalpathname(s))
OSError: [WinError 1] Incorrect function: '.'
>>> Path().absolute()
WindowsPath('R:/fileshare/fileshare-s')
>>> Path().absolute().resolve()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\zom\scoop\apps\miniconda3\current\envs\web\lib\pathlib.py", line 1159, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "C:\Users\zom\scoop\apps\miniconda3\current\envs\web\lib\pathlib.py", line 202, in resolve
    s = self._ext_to_normal(_getfinalpathname(s))
OSError: [WinError 1] Incorrect function: 'R:\\fileshare\\fileshare-s'

[eryksun]

When strict is "false", pathlib should not fail if the network
share is inaccessible.

A redirected filesystem (i.e. a share, whether local or remote) that's physically inaccessible should fail with a FileNotFoundError -- due to underlying errors such as ERROR_BAD_NETPATH (53) or ERROR_BAD_NET_NAME (67). This case is already handled by resolve() in non-strict mode. But error handling for other cases does need to be expanded.

A PermissionError should be ignored, from OS errors such as ERROR_ACCESS_DENIED (5), ERROR_SHARING_VIOLATION (32), and ERROR_NOT_READY (21).

It should ignore errors due to bad reparse points (e.g. filesystem symlinks and mount points), such as ERROR_INVALID_REPARSE_DATA (4392) and ERROR_REPARSE_TAG_INVALID (4393).

The resolve() method is supposed to raise a RuntimeError for a symlink loop (i.e. reparse loop). In Windows, the system detects this case as ERROR_CANT_RESOLVE_FILENAME (1921). It's not necessarily due to a reparse loop, but in practice it usually is. (It's actually due to the upper limit for the number of reparse attempts, which as of Windows 10 is no more than 63 attempts.) Maybe this error should be re-raised as a RuntimeError for the sake of compatibility with the POSIX implementation.

The above-mentioned cases are all due to WINAPI CreateFileW failing.
Additionally, the GetFinalPathNameByHandleW call in nt._getfinalpathname relies on several low-level system calls, any one of which can fail with ERROR_INVALID_FUNCTION (1) or ERROR_INVALID_PARAMETER (87) -- and possibly ERROR_NOT_SUPPORTED (50). These errors should be ignored in non-strict mode.

At a lower level, the reliability of nt._getfinalpathname could be improved for non-strict operation. It could fall back on other forms of the final name that may be supported. It could try FILE_NAME_OPENED instead of FILE_NAME_NORMALIZED to handle a filesystem that does not support normalizing 8.3 short names as long names. Try VOLUME_NAME_GUID instead of VOLUME_NAME_DOS to handle a volume that only has a GUID name (i.e. a volume device that's registered with the mount-point manager but lacks a DOS drive name or even a mountpoint on a volume that has a DOS drive name). Try VOLUME_NAME_NT to handle legacy volume/filesystem devices ('legacy' because they do not support the mount-point manager interface that was introduced almost 20 years ago in NT 5). For the sake of simplicity and performance, the latter case could be limited to well-known DOS device names such as r"\\?\PIPE" -> r"\Device\NamedPipe". The NT device path for the comparison should be queried instead of hard coded, e.g. via QueryDosDeviceW(L"PIPE", ...).

[christopherpickering]

Hi, is there a status update on this issue? Thanks!

[riffitos]

Still happening on Python 3.9.2 on Win10 when using a Ram Drive.

Any chance of getting this fixed?

[earonesty]

bug is worse than that:

perfectly valid redirected paths (winfsp ram drives for example) now break in python 3.9.6 (maybe fixed in later version?)

>>> import pathlib
>>> p=pathlib.Path('C:\\Users\\erik\\Atakama')
>>> p.resolve()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\users\erik\.pyenv\pyenv-win\versions\3.9.6\Lib\pathlib.py", line 1205, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "c:\users\erik\.pyenv\pyenv-win\versions\3.9.6\Lib\pathlib.py", line 206, in resolve
    s = self._ext_to_normal(_getfinalpathname(s))
OSError: [WinError 1005] The volume does not contain a recognized file system.

... yet....

dir 'C:\\Users\\erik\\Atakama'
Desktop.ini Personal\ Files Vault

mounted via winfsp

[eryksun]

perfectly valid redirected paths (winfsp ram drives for example)

I mounted a WinFsp MEMFS filesystem on a directory, which set a mountpoint that targets the root path of a volume device in the native "\Device" object directory. It didn't create a volume GUID name, which means the mountpoint manager isn't supported in this configuration.

The error code ERROR_UNRECOGNIZED_VOLUME (1005) is meaningless in this case. The mountpoint manager queries a volume with IOCTLs such as IOCTL_MOUNTDEV_QUERY_DEVICE_NAME, which the WinFsp virtual volume (in the above configuration) doesn't support. Weirdly, it returns the error code STATUS_UNRECOGNIZED_VOLUME instead of STATUS_INVALID_DEVICE_REQUEST. It does this as a lazy workaround for various IOCTLs it receives from filesystem drivers while the volume is in the process of being mounted [1][2]. The side effect is that it returns STATUS_UNRECOGNIZED_VOLUME for unhandled IOCTLs even when it's not getting mounted. This behavior should have been restricted to when the volume parameter block (VPB) is unmounted. Otherwise it should return the expected error code STATUS_INVALID_DEVICE_REQUEST (i.e. ERROR_INVALID_FUNCTION) instead of confusing users with a meaningless error.

WinFsp does support the mountpoint manager, in a restricted fashion. The mount target has to be a drive device name in the form "\.\X:". This gets registered with the mountpoint manager as the canonical DOS name of the volume. Since it's a global name, administrator access is required. It also creates a GUID volume name. Run mountvol.exe without arguments to find the volume name that's associated with the drive letter. Then run it again as mountvol <path> <volume_name>, where is an empty directory on which to mount the volume. Note that Python's os.path.realpath() will resolve the volume to the canonical drive name, even if the path traverses a directory mountpoint for the volume.

A new issue should be created to ignore ERROR_UNRECOGNIZED_VOLUME in 3.10+, for which Path.resolve() was updated to call os.path.realpath(). For 3.9, fixing Path.resolve() is still possible. There are 3 remaining bug releases planned: 3.9.9: (2022-01-03), 3.9.10 (2022-02-28), and 3.9.11 (2022-05-02).

---
[1] https://github.com/billziss-gh/winfsp/blob/v1.9/src/sys/devctl.c#L49
[2] winfsp/winfsp#177