expose the offset of a zipfile from the start of the file as a public API

[massimosala]

BPO	40301
Nosy	@Yhg1s, @stevendaprano, @serhiy-storchaka, @danifus, @massimosala
Files	py27_zipfile.patch: patch for Python 2.7.17

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-04-16.13:59:27.010>
labels = ['type-feature', 'library', '3.9']
title = 'zipfile module: new feature (two lines of code), useful for test, security and forensics'
updated_at = <Date 2020-04-18.14:35:44.170>
user = 'https://github.com/massimosala'

bugs.python.org fields:

activity = <Date 2020-04-18.14:35:44.170>
actor = 'massimosala'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-04-16.13:59:27.010>
creator = 'massimosala'
dependencies = []
files = ['49067']
hgrepos = []
issue_num = 40301
keywords = ['patch']
message_count = 13.0
messages = ['366597', '366601', '366685', '366689', '366690', '366695', '366709', '366711', '366712', '366713', '366714', '366717', '366720']
nosy_count = 6.0
nosy_names = ['twouters', 'alanmcintyre', 'steven.daprano', 'serhiy.storchaka', 'dhillier', 'massimosala']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40301'
versions = ['Python 3.9']

Linked PRs

• gh-84481: Add ZipFile.data_offset attribute #132165
• gh-84481: Make ZipFile.data_offset more robust #132178
• gh-84481: Remove the executable bits from test files #132507

[massimosala]

module zipfile

Tag "Components": I am not sure "Library (Lib)" is the correct one. If it isn't, please fix.

I use python to check zip files against malware.
In these files the are binary blobs outside the ZIP archive.
The malware payload isn't inside the ZIP file structure.
Example: a file "openme.zip" with this content :
[blob from offset 0 to offset 5678]
[ZIP archive from offset 5679 to end of file]

zipfile already handles this, finding the ZIP structure inside the file.

My change is just to add a new public property, to expose an internal variable: the file offset of the ZIP structure.

I know, I am after the code freeze of Python 2.7.18.
But the change is really trivial, see the diff.
I hope you can approve this patch for all the Python versions, also for 2.7, to have consistency. For 2.7 this is the last call.

[stevendaprano]

This is a new feature and cannot be added to older versions which are in feature-freeze.

Adding the feature to (say) Python 2.7.18 would be inconsistent, because it wouldn't exist in 2.7.0 through .17. Likewise for all the other versions before 3.9.

Personally, this sounds like a nice feature to have, and your use-case sounds convincing to me.

[massimosala]

Hi Steven

Every software "ecosystem" has its guidelines and I am a newbie about
python development.

Mmh I see your concerns. I agree about your deletions of all py 3 versions
before the latest 3.9.

About Py 2, I remark these facts:

• there are a lot of forensics tools still written for py 2;
• python 2.7.18 will be forever the last python 2 and I think is it fine to
  end-users to have zipfile with this feature both in py 2.7 and py 3.9;
• in the code there isn't any new routine to test: the change is just to
  expose one internal variable.

I agree my request is an exception but I think you have to agree this
situation is exceptional.
IMHO rules must exist to help us and I think this request doesn't carry any
burden.

I ask you please

• to reconsider my request
• anyway, to put me in contact with zipfile mainteners, I don't know how to
  reach them but I want to hear them about this.

Many thanks, Massimo

[danifus]

Could something similar be achieved by looking for the earliest file header offset?

def find_earliest_header_offset(zf):
    earliest_offset = None
    for zinfo in zf.infolist():
        if earliest_offset is None:
            earliest_offset = zinfo.header_offset
        else:
            earliest_offset = min(zinfo.header_offset, earliest_offset)
    return earliest_offset

You could also adapt this using

zinfo.compress_size + len(zinfo.FileHeader())

to see if there were any sections inside the archive which were not referenced from the central directory. Not sure if zip files with arbitrary bytes inside the archive would be valid everywhere, but I think they are using zipfile.

You can also have zipped content inside an archive which has a valid fileheader but no reference from the central directory. Those entries are discoverable by implementations which process content serially from the start of the file but not implementations which rely on the central directory.