IDLE: Don't freeze Shell when user enters sys.stdout.shell.console

[HarrisonChudleigh]

BPO	34708
Nosy	@terryjreedy, @iritkatriel
Files	fail.py: After running this file, IDLE hangs or crashes, depending on version.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = None
created_at = <Date 2018-09-17.02:53:01.502>
labels = ['3.10', 'expert-IDLE', '3.9', 'type-crash', '3.11']
title = 'Odd crashes/freezes when sys.stdout.shell.console is typed'
updated_at = <Date 2021-10-18.21:41:12.838>
user = 'https://bugs.python.org/HarrisonChudleigh'

bugs.python.org fields:

activity = <Date 2021-10-18.21:41:12.838>
actor = 'iritkatriel'
assignee = 'terry.reedy'
closed = False
closed_date = None
closer = None
components = ['IDLE']
creation = <Date 2018-09-17.02:53:01.502>
creator = 'Harrison Chudleigh'
dependencies = []
files = ['47808']
hgrepos = []
issue_num = 34708
keywords = []
message_count = 5.0
messages = ['325509', '325510', '325569', '404215', '404224']
nosy_count = 3.0
nosy_names = ['terry.reedy', 'Harrison Chudleigh', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue34708'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-78889: Stop IDLE Shell freezes from sys.stdout.shell.xyz #121876
• [3.13] gh-78889: Stop IDLE Shell freezes from sys.stdout.shell.xyz (GH-121876) #121911
• [3.12] gh-78889: Stop IDLE Shell freezes from sys.stdout.shell.xyz (GH-121876) #121912

[HarrisonChudleigh]

If sys is imported and sys.stdout.shell.console is typed (this is what fail.py does TJR), IDLE does not return to the prompt. Ctrl-C has no effect and after a restart with Ctrl-F6, IDLE does not react after a command is typed in. This also occurs when other variables such as sys.stdout.shell.stdout are used, but not with non-existent variables such as sys.stdout.shell.not_a_variable. No such issue occurs using the command line; however, sys.stdout.shell is not defined in the command line. This behaviour exists in at least Python 3.6.1 and 3.7.0 on Mac OS X 10.9.

This might be related to a similar issue that occurs in at least Python 3.2 on Windows 2000. Entering sys.stdout.shell.console causes IDLE to crash immediately; a crash also occurs when you type sys.stdout.shell.console. and wait for IDLE to bring up the list of attributes.

I know that this bug shouldn't be encountered unless you're using the IDLE PyShell object for some reason, but it still seems weird that getting the value of a variable causes crashes and hangs, especially when I know that the variable exists from looking at idlelib.

This is marked as Python 3.4 to 3.7 because I tested with 3.7, 3.6 and 3.2 and I don't think that this would have been fixed and then broken again. I don't know about 3.8, though.

[HarrisonChudleigh]

Edit: I checked Python 3.4 (Windows XP SP3). It hangs in the same way as Python 3.6/3.7.

[terryjreedy]

3.4 and 3.5 only get security fixes and I doubt this qualifies.

I reproduced the described behavior with 3.7 on Win 10, so it is not Mac-specific.

Python sys.stdout: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
IDLE's sys.stdout: <idlelib.run.PseudoOutputFile object at 0x00000246821B1E10>
The PseudoFiles are unusual in being internal objects accessible from user code.

Comparing attributes with public names (not starting with single or double underscore) PseudoOutputFile lacks buffer, line_buffering, mode, reconfigure, and write_through. bpo-21995 is about fixing or documenting each omission for all 3 streams.

PseudoFiles add attributes 'shell' and 'tags'. Both should have private names that warn users that their behavior is undocumented and usage is at one's own risk. 'shell' should be double underscored. This minimal fix would be sufficient to close this issue. Some manual testing should be sufficient for such a change.

tags should be singular, as it is one of 'stdin', 'stdout', or 'stderr', and renamed stream. The tag is used in the write method to tell Shell whether to display with the configured stdout or stderr colors.

shell is an idlelib.rpc.RPCProxy object. As an object in itself, in the run process, it has 2 public attributes: oid ('console') and sockio (an idlelib.run.MyHandler object). But as a proxy representing the Shell console object in the IDLE process, it supposedly 'has' all the attributes of the latter. These are accessed with a custom __getattr__.

    def __getattr__(self, name):
        if self.__methods is None:
            self.__getmethods()
        if self.__methods.get(name):
            return MethodProxy(self.sockio, self.oid, name)
        if self.__attributes is None:
            self.__getattributes()
        if name in self.__attributes:
            value = self.sockio.remotecall(self.oid, '__getattribute__',
                                           (name,), {})
            return value
        else:
            raise AttributeError(name)

Note that self.__attributes is mangled to shell._RPCProxy__attributes when accessed externally. (Also, it is a set still implemented in this ancient code as a dict with int 1 values.)

More importantly, remote calls to the Shell console can only pass and return objects that can be pickled, and there is currently no provision for graceful failure. Hence sys.stdout.shell.width is (for me, currently) 80, but .console or .stdout are impossible to return.

I am extremely dubious about trying to fix this in the IDLE side of the link. Instead, the proxy should be initialized with sets of known usable attributes and methods. I think the underlying problem is trying to be unnecessarily and impossibly generic. Run only calls the .readline, .write, and .close methods of the console proxy. It only calls stack viewer setup on the interp proxy. A third proxy, for flist, is passed to the stack viewer (I have not yet checked what is needed for that). Restricting proxies to things that are needed and *should* work would make it more feasible to test them. But setting up automated tests would still require some work.

[iritkatriel]

Terry, I don't see any PseudoOutputFile in the current codebase. Is this out of date?

[terryjreedy]

Unfortunately, no. The standard stream replacements were renamed to StdOutputFile, etc, and slightly revised, but the buggy behavior remains. The worst is that shell restart does not completely restart the shell. It has to be closed instead.

I would like to fix this first by adding 'provision for graceful failure'. 'try' is now nearly free when there is no exception.