ZipFile.extractall fails if zipfile.Path object is created based on the ZipFile object

[mmohrhard]

Bug report

The following code works fine with Python 3.8 and 3.9 but starts failing with Python 3.10.

It seems that zipfile.Path creates an entry in the zipfile.ZipFile object that does not exist in the underlying file and therefore makes zipfile.ZipFile.extractall fail.

import zipfile
import tempfile

path = tempfile.mktemp()

f = zipfile.ZipFile(path, "w")
f.writestr("data/test.txt", "test1")
f.writestr("data/test2.txt", "test2")
f.close()

dest = tempfile.mkdtemp()

p = zipfile.ZipFile(path)
src_zipfile_path = zipfile.Path(p, "data")
p.extractall(dest)

Your environment

Working version:

(python39) [moggi@mars cloudfluid]$ python --version
Python 3.9.16
(python39) [moggi@mars cloudfluid]$ python test.py
(python39) [moggi@mars cloudfluid]$ 

Failing versions:

3.10.9:

(python_310_2) [moggi@mars cloudfluid]$ python --version
Python 3.10.9
(python_310_2) [moggi@mars cloudfluid]$ python test.py
Traceback (most recent call last):
  File "/home/moggi/devel/cloudfluid/test.py", line 15, in <module>
    p.extractall(dest)
  File "/home/moggi/devel/envs/python_310_2/lib/python3.10/zipfile.py", line 1645, in extractall
    self._extract_member(zipinfo, path, pwd)
  File "/home/moggi/devel/envs/python_310_2/lib/python3.10/zipfile.py", line 1667, in _extract_member
    member = self.getinfo(member)
  File "/home/moggi/devel/envs/python_310_2/lib/python3.10/zipfile.py", line 1441, in getinfo
    raise KeyError(
KeyError: "There is no item named 'data/' in the archive"

and 3.11.0

(python311) [moggi@mars cloudfluid]$ python --version
Python 3.11.0
(python311) [moggi@mars cloudfluid]$ python test.py
Traceback (most recent call last):
  File "/home/moggi/devel/cloudfluid/test.py", line 15, in <module>
    p.extractall(dest)
  File "/home/moggi/devel/envs/python311/lib/python3.11/zipfile.py", line 1677, in extractall
    self._extract_member(zipinfo, path, pwd)
  File "/home/moggi/devel/envs/python311/lib/python3.11/zipfile.py", line 1699, in _extract_member
    member = self.getinfo(member)
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/moggi/devel/envs/python311/lib/python3.11/zipfile.py", line 1473, in getinfo
    raise KeyError(
KeyError: "There is no item named 'data/' in the archive"

If this is the expected new behavior I think the documentation for 3.10 should mention this breaking change and the zipfile.Path documentation might need an entry about this problem.

The zipfile.Path object is used in the original code to check if the "data/" directory exists in the zip file.

Linked PRs

• gh-101566: Sync with zipp 3.14. #102018
• [3.11] gh-101566: Sync with zipp 3.14. (GH-102018). #102090
• [3.10] gh-101566: Sync with zipp 3.14. (GH-102018). #102091

[arhadthedev]

@serhiy-storchaka, @Yhg1s, @gpshead as zipfile experts.

[gpshead]

@jaraco

[jaraco]

This issue occurs because of the optimizations / tradeoffs made in attempting to extend an open file object. Here's what's going on:

• zipfile.Path needs to be able to infer directories when there are none in the zipfile (i.e. the presence of data/test.txt implies the presence of data/ even though it's not explicitly a member of the zipfile).
• To accomplish this goal, the CompleteDirs wraps the original zipfile.
• Due to challenges with the lifecycle of a zipfile (which may have been created in memory or exist on disk as an open file), wrapping a Zipfile in a Path actually mutates its class, giving the original Zipfile object the CompleteDirs behavior.
• In extractall, it's assumed that every item in the namelist is also a member, but with CompleteDirs, that's no longer true.

This report is the first I've seen that this (admittedly impure) behavior has caused any trouble.

I agree that at the very least, it's worthwhile documenting these limitations.

I did try to track down the origins of the change that led to this regression in behavior. I took a brief look at the readme for zipp, the forward/backport, and it doesn't mention any changes for Python 3.10 (now updated to reflect new findings).

Okay. I see the change occurred in ebbe803, bpo-40564/#84744. That bug pretty thoroughly describes the thought process and the tradeoffs considered.

I can think of a few ways to address the issue.

• document it as unsupported behavior
• document it as unsupported behavior, but provide a mechanism to restore the original object
• add support for zipfile.Zipfile to be lenient to dirs not present
• add support to CompleteDirs to return a virtual member info for an implied dir
• CompleteDirs to override extractall to bypass the issue

I'm leaning toward (3) or (4). (4) has an advantage over all of the other options in that, if it can be implemented, it provides better compatibility for more use cases (not just ZipFile.extractall).

[jaraco]

In jaraco/zipp#90, I've drafted a patch implementing option 4, to be released as zipp 3.12.1. I'd like to get some feedback on the concept before porting that to cpython.