Add underscore as a decimal separator for string formatting

[TerryDavis]

BPO	43624
Nosy	@rhettinger, @mdickinson, @vstinner, @ericvsmith, @serhiy-storchaka, @domdfcoding

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-03-25.17:19:07.475>
labels = ['interpreter-core', 'type-feature', '3.11']
title = 'Add underscore as a decimal separator for string formatting'
updated_at = <Date 2021-05-04.15:37:03.108>
user = 'https://bugs.python.org/TerryDavis'

bugs.python.org fields:

activity = <Date 2021-05-04.15:37:03.108>
actor = 'Terry Davis'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2021-03-25.17:19:07.475>
creator = 'Terry Davis'
dependencies = []
files = []
hgrepos = []
issue_num = 43624
keywords = []
message_count = 17.0
messages = ['389508', '389512', '389517', '389529', '389534', '389546', '389547', '389574', '389687', '389708', '389709', '389735', '389736', '389744', '389754', '389762', '392911']
nosy_count = 7.0
nosy_names = ['rhettinger', 'mark.dickinson', 'vstinner', 'eric.smith', 'serhiy.storchaka', 'Terry Davis', 'domdfcoding']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue43624'
versions = ['Python 3.11']

Linked PRs

• gh-87790: support thousands separators for formatting fractional part of floats #125304
• gh-87790: support thousands separators for formatting fractional part of Decimal's #132202
• gh-87790: support thousands separators for formatting fractional part of Fraction's #132204
• gh-87790: fix precision_with_grouping in formatting mini-language #132205

[TerryDavis]

Proposal:
Enable this
>>> format(12_34_56.12_34_56, '_._f')
'123_456.123_456'

Where now only this is possible
>>> format(12_34_56.12_34_56, '_.f')
'123_456.123456'

Based on the discussion in the Ideas forum, three core devs support this addition.
https://discuss.python.org/t/add-underscore-as-a-thousandths-separator-for-string-formatting/7407

I'm willing to give this a try if someone points me to where to add tests and where the float formatting code is. This would be my first CPython contribution.

The feature freeze for 3.10 is 2021-05-03.
https://www.python.org/dev/peps/pep-0619/#id5

[rhettinger]

IIRC there is ISO recommending that after the decimal point, digits be arranged in groups of five. I think is also how printed reference tables are typically formatted.

[rhettinger]

Some brief research
===================

""" in numbers four or more digits long, use commas to set off groups of three digits, counting leftward from the decimal point, in the standard American style. For long decimal numbers, do not use any digit-group separators to the right of the decimal point."""
— Google Style Guide https://developers.google.com/style/numbers

The CRC math handbook uses groups of five after the decimal point.
See §1.2.4 in
http://dl.icdst.org/pdfs/files/2a2cbcfc89598fd83c315ce45c1ee663.pdf

NIST Guide for using SI units: """The digits of numerical values having more than four digits on either side of the decimal marker are separated into groups of three using a thin, fixed space counting from both the left and right of the decimal marker. For example, 15 739.012 53 is highly preferred to 15739.01253. Commas are not used to separate digits into groups of three. (See Sec. 10.5.3.)"""
— page vi in https://physics.nist.gov/cuu/pdf/sp811.pdf#10.5.2

StackExchange question on the topic:
https://math.stackexchange.com/questions/182775/convention-of-digit-grouping-after-decimal-point

The important reference, ISO 80000:1 discusses this in section 7, "Printing rules", but the standard is not publicly available.

[domdfcoding]

ISO 80000-1:2009 recommends groups of three digits either side of the decimal sign.

[ericvsmith]

If we do anything for float, we should do the same for decimal.Decimal.

[vstinner]

If we do anything for float, we should do the same for decimal.Decimal.

and complex ;-)

[vstinner]

How backward incompatible and annoying would it be to modify the behavior of the existing "_f" format?

Do you see use cases which only want to group digits in the integer part but not the fractional part?

According to https://discuss.python.org/t/add-underscore-as-a-thousandths-separator-for-string-formatting/7407 discussion, grouping digits was first designed for integers, and the fractional part of floats was simply ignored/forgotten. I mean, it doesn't sound like a deliberate choice to not group digits in the fractional part.

The advantage of changing "_f" format is to keep backward compatibility: Python 3.9 and older would not group digits in the fractional part, but at least they don't fail with an error. If you write code with "_._f" format, you need a fallback code path for Python 3.9 and older:

if sys.version_info >= (3, 10):
   text = f"my {...} very {...} long {...} and {...} complex {...} format string: x={x:_._f}"
else:
   text = f"my {...} very {...} long {...} and {...} complex {...} format string: x={x:_f}"

Or:

text = f"my {...} very {...} long {...} and {...} complex {...} format string:" + (f"x={x:_f}" if sys.version_info >= (3, 10) else "x={x:_f}")

Or many other variants.

The main drawback is the risk to break tests relying on the exact output.

About the separator character and the number of digits per group, IMO there is no standard working in all countries and all languages. But since we have a strict rule of 3 digits with "_" separator, I am fine with doing the same for the fractional part. It's an "arbitrary" choice, but at least, it's consistent.

People wanting a different format per locale/language should write their own function. Once enough people will agree on such API, we can consider to add it to the stdlib. But for now, IMO 3 digits with "_" is good enough.

By the way, I agree that it's hard to read numbers with many digits in the decimal part ;-)

>>> f"{1/7:_.30f}"
'0.142857142857142849212692681249'

>>> f"{10**10+1/7:_.10f}"
'10_000_000_000.1428565979'

[TerryDavis]

Good point Victor, though I wonder how likely it is that a person using 3.10 would only use this particular new feature, and have an otherwise backwards-compatible codebase.

This isn't something that I asked about out of necessity, and there hasn't been any other discussion of this idea that anyone can remember.

On the other hand, I suppose it would be possible to have a feature flag that can be used to disable decimal underscores in 3.10 to prevent test failures. Just spitballing...

[vstinner]

On the other hand, I suppose it would be possible to have a feature flag that can be used to disable decimal underscores in 3.10 to prevent test failures. Just spitballing...

I wrote PEP-606 -- Python Compatibility Version https://www.python.org/dev/peps/pep-0606/ and it was rejected.