test_tarfile fails on 32 bit systems in NoneInfoExtractTests_Data

[kanavin]

These tests are newly added, and they produce the following fails on 32 bit x86 systems (running python 3.11.4):

======================================================================
ERROR: test_extractall_none_gid (test.test_tarfile.NoneInfoExtractTests_Data.test_extractall_none_gid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3067, in test_extractall_none_gid
    with self.extract_with_none('gid'):
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3026, in extract_with_none
    self.tar.extractall(DIR, filter='fully_trusted')
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: test_extractall_none_gname (test.test_tarfile.NoneInfoExtractTests_Data.test_extractall_none_gname)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3075, in test_extractall_none_gname
    with self.extract_with_none('gname'):
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3026, in extract_with_none
    self.tar.extractall(DIR, filter='fully_trusted')
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: test_extractall_none_mode (test.test_tarfile.NoneInfoExtractTests_Data.test_extractall_none_mode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3053, in test_extractall_none_mode
    with self.extract_with_none('mode') as DIR:
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3026, in extract_with_none
    self.tar.extractall(DIR, filter='fully_trusted')
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: test_extractall_none_mtime (test.test_tarfile.NoneInfoExtractTests_Data.test_extractall_none_mtime)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3034, in test_extractall_none_mtime
    with self.extract_with_none('mtime') as DIR:
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3026, in extract_with_none
    self.tar.extractall(DIR, filter='fully_trusted')
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: test_extractall_none_uid (test.test_tarfile.NoneInfoExtractTests_Data.test_extractall_none_uid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3063, in test_extractall_none_uid
    with self.extract_with_none('uid'):
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3026, in extract_with_none
    self.tar.extractall(DIR, filter='fully_trusted')
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: gid is greater than maximum

======================================================================
ERROR: test_extractall_none_uname (test.test_tarfile.NoneInfoExtractTests_Data.test_extractall_none_uname)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3071, in test_extractall_none_uname
    with self.extract_with_none('uname'):
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3026, in extract_with_none
    self.tar.extractall(DIR, filter='fully_trusted')
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_Default)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3002, in setUpClass
    tar.extractall(cls.control_dir, filter=cls.extraction_filter)
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_FullyTrusted)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3002, in setUpClass
    tar.extractall(cls.control_dir, filter=cls.extraction_filter)
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

======================================================================
ERROR: setUpClass (test.test_tarfile.NoneInfoExtractTests_Tar)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.11/test/test_tarfile.py", line 3002, in setUpClass
    tar.extractall(cls.control_dir, filter=cls.extraction_filter)
  File "/usr/lib/python3.11/tarfile.py", line 2257, in extractall
    self._extract_one(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.11/tarfile.py", line 2320, in _extract_one
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.11/tarfile.py", line 2418, in _extract_member
    self.chown(tarinfo, targetpath, numeric_owner)
  File "/usr/lib/python3.11/tarfile.py", line 2546, in chown
    os.chown(targetpath, u, g)
OverflowError: uid is greater than maximum

----------------------------------------------------------------------
Ran 545 tests in 8.309s

FAILED (errors=9, skipped=8)

I believe the file they are tripping over is:

alex@Zen2:/srv/work/alex/cpython$ tar --list --numeric-owner -vf Lib/test/testtar.tar|grep regtype-gnu-uid
-rw-r--r-- 4294967295/4294967295 7011 2003-01-06 00:19 gnu/regtype-gnu-uid

Linked PRs

• gh-107811: tarfile: treat overflow in UID/GID as failure to set it #108369

[encukou]

What OS is this? I tried on 32-bit Debian but could not reproduce.

Looks like the tests uncovered a real issue on systems with a small uid_t -- extraction should behave like when chmod fails in any other way.

[kanavin]

What OS is this? I tried on 32-bit Debian but could not reproduce.

Looks like the tests uncovered a real issue on systems with a small uid_t -- extraction behave like when chmod fails in any other way.

It's Yocto's poky using qemu x86 target. I can run local tests to help diagnose this if needed.

[encukou]

Can you try out this patch?

diff --git a/Lib/tarfile.py b/Lib/tarfile.py
index d62323715b4..5ef2394ad3a 100755
--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -2557,7 +2557,8 @@ def chown(self, tarinfo, targetpath, numeric_owner):
                     os.lchown(targetpath, u, g)
                 else:
                     os.chown(targetpath, u, g)
-            except OSError as e:
+            except (OSError, OverflowError) as e:
+                # OverflowError can be raised if an ID doesn't fit in `id_t`
                 raise ExtractError("could not change owner") from e
 
     def chmod(self, tarinfo, targetpath):

[kanavin]

Can you try out this patch?

With the patch the test is passing:

root@qemux86:/usr/lib/python3.11# python3 -m test test_tarfile
0:00:00 load avg: 0.28 Run tests sequentially
0:00:00 load avg: 0.28 [1/1] test_tarfile

== Tests result: SUCCESS ==

1 test OK.

Total duration: 10.2 sec
Tests result: SUCCESS

[encukou]

Merged. Thanks for the report and testing!
As this is not a supported architecture, and it's only come up because the tests are now more comprehensive, I don't think it should be backported to 3.12.0rc. I assume poky has ways to patch it in if needed.

[kanavin]

No problem. I'll do a local backport.