Duplicate entry in 'Objects/unicodetype_db.h'

[LiarPrincess]

BPO	47243
Nosy	@vstinner, @ezio-melotti, @LiarPrincess
PRs	bpo-47243: Duplicate entry in 'Objects/unicodetype_db.h' #32376

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-04-06.17:23:31.334>
labels = ['type-feature', 'expert-unicode', '3.11']
title = "Duplicate entry in 'Objects/unicodetype_db.h'"
updated_at = <Date 2022-04-06.17:48:43.752>
user = 'https://github.com/LiarPrincess'

bugs.python.org fields:

activity = <Date 2022-04-06.17:48:43.752>
actor = 'LiarPrincess'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2022-04-06.17:23:31.334>
creator = 'LiarPrincess'
dependencies = []
files = []
hgrepos = []
issue_num = 47243
keywords = ['patch']
message_count = 2.0
messages = ['416889', '416892']
nosy_count = 3.0
nosy_names = ['vstinner', 'ezio.melotti', 'LiarPrincess']
pr_nums = ['32376']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue47243'
versions = ['Python 3.11']

[LiarPrincess]

This one is so tiny that I'm not really sure we want to merge it…

=== Problem ===

[Objects/unicodetype_db.h](https://github.com/python/cpython/blob/main/Objects/unicodetype_db.h) starts in a following way:

/* a list of unique character type descriptors */
const _PyUnicode_TypeRecord _PyUnicode_TypeRecords[] = {
    {0, 0, 0, 0, 0, 0},
    {0, 0, 0, 0, 0, 0},
    {0, 0, 0, 0, 0, 32},
    {0, 0, 0, 0, 0, 48},
    …

The 1st record ({0, 0, 0, 0, 0, 0}) is duplicated.
This is not a problem, since the 1st occurrence is never used, but if we wanted to remove it then this is the ticket about it.

=== Detailed description ===

[Objects/unicodetype_db.h](https://github.com/python/cpython/blob/main/Objects/unicodetype_db.h) is generated by [Tools/unicode/makeunicodedata.py](https://github.com/python/cpython/blob/main/Tools/unicode/makeunicodedata.py) (I removed irrelevant lines):

def makeunicodetype(unicode, trace):
    dummy = (0, 0, 0, 0, 0, 0)
    table = [dummy] # (1)
    cache = {0: dummy} # (2)

    for char in unicode.chars:
        # Things…

        item = (upper, lower, title, decimal, digit, flags)

        i = cache.get(item) # (3)
        if i is None:
            cache[item] = i = len(table)
            table.append(item)

        index[char] = i

• (1) - list which contains unique character properties (as (upper, lower, title, decimal, digit, flags) tuples)
• (2) - mapping from character properties to index in table - improperly initialized as a mapping from index to character properties
• (3) - we check if the current tuple is in cache

=== Result ===

The first time we get to a character that has (0, 0, 0, 0, 0, 0) properties (which is code point 0 - NULL) we check if it is in cache. It it not (there is an entry that goes from index 0 to (0, 0, 0, 0, 0, 0) - the other way around), so we add this entry to table and cache.

=== Fix ===

In the line (2) we should have: cache = {dummy: 0}. Obviously after doing so we have to run makeunicodedata.py - this is why this simple change modifies a lot of lines.

I will submit PR on github in just a sec…

[LiarPrincess]

CLA is signed, but there is this 'it might take a few days before your tracker profile is updated'.

Added version 3.11 (present also in previous versions, bot no point in back-porting it).

Github: #32376

[hauntsaninja]

Thanks, looks like this has been completed