tokenize.py emits spurious NEWLINE if file ends on a comment without a newline

[aroberge]

Bug report

This issue, which was reported in https://bugs.python.org/issue35107#msg328884 for much older Python versions in the case where a file ended on a comment, has resurfaced for Python 3.12.

Python 3.12.0b2 (tags/v3.12.0b2:e6c0efa, Jun  6 2023, 15:37:48) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tokenize
>>> import io
>>> tokenize.untokenize(tokenize.generate_tokens(io.StringIO('#').readline))
'#\n'

Note that the problem also arises in the following example:

>>> tokenize.untokenize(tokenize.generate_tokens(io.StringIO('a\n ').readline))
'a\n \n'

Linked PRs

• gh-105435: Fix spurious NEWLINE token if file ends with comment without a newline #105442
• [3.12] gh-105435: Fix spurious NEWLINE token if file ends with comment without a newline (GH-105442) #105444

[AlexWaygood]

This looks similar to #105259 — does the problem still reproduce with 3.12.0b2?

(Cc. @pablogsal / @lysnikolaou)

EDIT: I see you're using 3.12.0b2 already. Apologies.

[pablogsal]

CC: @mgmacias95