Token lines in the tokenize module always end in new line

[pablogsal]

As a consequence of the changes in the Python tokenizer post PEP701, now the line attribute always ends in a new line character. We should remove this as it adds no information and wasn't there before.

Linked PRs

• gh-104825: Remove implicit newline in the line attribute in tokens emitted in the tokenize module #104846
• [3.12] gh-104825: Remove implicit newline in the line attribute in tokens emitted in the tokenize module (GH-104846). #104850
• [3.12] gh-104825: add omitted idlelib text fix #104880

[lysnikolaou]

Yeah, I ran across that while working on #104798. Are you already working on a fix or shall I take it up?

[pablogsal]

Yeah, I ran across that while working on #104798. Are you already working on a fix or shall I take it up?

I am working on a fix, should be up in half an hour or so

[lysnikolaou]

#104846 now removes a newline, even when a newline exists in the original source.

❯ ./python.exe
Python 3.13.0a0 (heads/setuptools-peg-generator-dirty:c7503656c2, May 24 2023, 12:24:20) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tokenize
>>> import io
>>> file = io.StringIO('1 + 2 + 3\n')
>>> for token in tokenize.generate_tokens(file.readline):
...     print(token)
... 
TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1 + 2 + 3')
TokenInfo(type=55 (OP), string='+', start=(1, 2), end=(1, 3), line='1 + 2 + 3')
TokenInfo(type=2 (NUMBER), string='2', start=(1, 4), end=(1, 5), line='1 + 2 + 3')
TokenInfo(type=55 (OP), string='+', start=(1, 6), end=(1, 7), line='1 + 2 + 3')
TokenInfo(type=2 (NUMBER), string='3', start=(1, 8), end=(1, 9), line='1 + 2 + 3')
TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 9), end=(1, 10), line='1 + 2 + 3')
TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')

[lysnikolaou]

@pablogsal Do you think the above should be fixed? Will you work on it?

[pablogsal]

@pablogsal Do you think the above should be fixed? Will you work on it?

Hummmmm, not sure how easy is going to be for us to distinguish between implicit and explicit newlines at the end. I would say, leave it like is now unless you disagree, in which case let's see if we can fix it easily.