input function truncates prompt by NULL byte

[ghost]

BPO	30431
Nosy	@terryjreedy, @serhiy-storchaka, @kushaldas, @eryksun
PRs	gh-74616: Raise ValueError in case of NULL in input prompt #1738

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/kushaldas'
closed_at = None
created_at = <Date 2017-05-22.15:09:04.110>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.10']
title = 'input function truncates prompt by NULL byte'
updated_at = <Date 2021-03-04.19:49:59.064>
user = None

bugs.python.org fields:

activity = <Date 2021-03-04.19:49:59.064>
actor = 'eryksun'
assignee = 'kushal.das'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2017-05-22.15:09:04.110>
creator = '\xd0\x9a\xd0\xbe\xd1\x81\xd1\x82\xd1\x8f \xd0\xa7\xd0\xbe\xd0\xbb\xd0\xb0\xd0\xba'
dependencies = []
files = []
hgrepos = []
issue_num = 30431
keywords = []
message_count = 7.0
messages = ['294153', '294176', '294179', '294589', '294595', '294612', '294630']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'serhiy.storchaka', 'kushal.das', 'eryksun', '\xd0\x9a\xd0\xbe\xd1\x81\xd1\x82\xd1\x8f \xd0\xa7\xd0\xbe\xd0\xbb\xd0\xb0\xd0\xba']
pr_nums = ['1738']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30431'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[ghost]

input('some\x00 text')
will prompt for `some` instead of `some text`

[kushaldas]

I am picking this up.

[eryksun]

PyOS_Readline uses a null-terminated prompt string. So input() needs to raise a ValueError if the prompt contains null characters, e.g. if PyBytes_GET_SIZE(po) != strlen(promptstr).

[terryjreedy]

Experiments running 3.6.1 on Windows in console:

python -c "print('some\0 text')
some text # \0 printed as ' ', as determined by
# >python -c "for c in 'some text': print(ord(c))" printing 32 32

python -c "input('some\0 text')
some

In IDLE, both print full string with actual null byte. As a result, attempting the ord() test above generates "SyntaxError: source code string cannot contain null bytes". Cutting the output from IDLE and pasting here (FireFox) results in a truncated 'some'.

Conclusions:

1. Python is emitting \0 to stdout. That is what python should do when asked to, as documented.
2. Currently, if one wants to prompt with strings containing \0, use IDLE or a similar GUI-based shell.
3. input() should *not* reject prompts with \0
4. If \0 is a problem for a particular stdout, its handler could raise ValueError, or replace \0 with r'\0', or replace \0 with ' ' (as with print to Widows console.
5. When running in Windows console, the prompt output part of input(prompt) should treat \0 the same as print(prompt) does. I am surprised it does not, as "input(prompt)" has been described as shorthand for "print(prompt, end=''); input()"

[terryjreedy]

As near as I can tell, the patch introduces a regression. The doc for input(prompt) says "If the prompt argument is present, it is written to standard output without a trailing newline." The handling of the prompt is up to stdout.write and thence physical displays devices. Python does not specify what happens with any particular character, including \r, \t, and astral chars.

C:\Users\Terry>python -c "import sys; sys.stdout.write('some\0 text\n')
some text

The bug on Windows is that input('some\0 text\n') does not produce the same display.

[eryksun]

The solution to raise a ValueError for the PyOS_Readline case won't change the behavior of input() in cases where the prompt is written to sys.stdout.

As to copying what print() would do, I guess it's possible to fake it for common platforms. You'd have to process the prompt string to emulate the platform behavior, which commonly means either ignoring '\0' or replacing it with space. However, this can't emulate raw console or terminal modes that actually write '\0'.

A Linux terminal ignores '\0' characters written to it, e.g. "some\0text" prints as "sometext". I didn't look into modifying the terminal settings, but it wouldn't surprise me if different behavior is possible.

The Windows console replaces '\0' with a space when the screen buffer is in the default cooked mode (i.e. ENABLE_PROCESSED_OUTPUT). For raw output it writes '\0' to the screen buffer, which displays like a space, but copy and paste won't work as expected.

I am surprised it does not, as "input(prompt)" has been
described as shorthand for "print(prompt, end=''); input()"

That's how the fallback case works, such as in IDLE. In the readline case, it doesn't even write the prompt to C stdout. It gets written to stderr...

For Windows, if stderr is a console screen buffer, it transcodes the prompt from UTF-8 to UTF-16 and writes it via WriteConsoleW. Otherwise it calls fprintf to write it to stderr.

[serhiy-storchaka]

If stdin and stdout are attached to a terminal, input() uses the readline library (if it is available) for displaying the prompt and reading user input. This is the limitation of the readline library and Python C API that this works with null-terminated C strings. You also can't enter the null character.

UnicodeEncodingError is raised if the prompt contains characters not encodable with the stdout encoding. Consider ValueError raised if the prompt contains the null character as similar restriction.