Pickletools Default Encoding

[Legoclones]

Bug report

Bug description:

When unpickling using _pickle.c or pickle.py through load/loads, an encoding can be specified using the encoding argument, with the default being ASCII. However, pickletools does not support custom encodings and instead makes assumptions about what encoding it uses, which can lead to either incorrect data being displayed or erroring/not erroring when the normal unpickling process would error.

The three opcodes that I have found this in are STRING, BINSTRING, and SHORT_BINSTRING:

• On line 359, it assumes ASCII for STRING
• On line 456, it assumes latin-1 for BINSTRING
• On line 422, it assumes latin-1 for SHORT_BINSTRING

I think the best solution would be to support encodings as an optional argument in pickletools.py, with the default being set to ASCII (since that's the default encoding for pickle.py and _pickle.c).

CPython versions tested on:

3.11

Operating systems tested on:

Linux

Linked PRs

• gh-126997: Fix support of non-ASCII strings in pickletools #127062
• [3.13] gh-126997: Fix support of non-ASCII strings in pickletools (GH-127062) #127094
• [3.12] gh-126997: Fix support of non-ASCII strings in pickletools (GH-127062) #127095

[kanishka-coder0809]

hey @Legoclones I think I can work with this issue ...can you please assign me with this

[Legoclones]

I don't think I have permission to assign someone

[kanishka-coder0809]

So should I start working upon this and thank make a PR.?

[serhiy-storchaka]

It is not a problem if pickletools accepts an input which is not valid for pickle. But there are real bugs here:

• pickletools does not support STRING with non-ASCII argument. pickle allows to specify encoding (ASCII by default).
• pickletools does not support GLOBAL with non-ASCII argument. pickle uses UTF-8.
• pickletools.dis() outputs non-ASCII arguments for BINSTRING and SHORT_BINSTRING as Latin1-encoded. pickle allows to specify encoding (ASCII by default). It would be better to output non-ASCII bytes as \xXX.

#127062 fixes them.

[serhiy-storchaka]

@kanishka-coder0809, sorry, I already worked on this issue. It is slightly larger than initially was reported. On other hand, I think that I found a better solution than supporting optional argument for encoding. The latter would require more significant rewriting of the pickletools module and likely could be too complex for backporting.

[Legoclones]

That looks great!