Regex \B doesn't match empty string

[Alcaro]

Bug report

Bug description:

>>> import re
>>> list(re.finditer(r'\b', 'e'))
[<re.Match object; span=(0, 0), match=''>, <re.Match object; span=(1, 1), match=''>]
>>> list(re.finditer(r'\B', 'e'))
[]
>>> list(re.finditer(r'\b', '%'))
[]
>>> list(re.finditer(r'\B', '%'))
[<re.Match object; span=(0, 0), match=''>, <re.Match object; span=(1, 1), match=''>]
>>> list(re.finditer(r'\b', ''))
[]
>>> list(re.finditer(r'\B', ''))
[]

Apparently the empty string neither is nor isn't a word boundary. Is that supposed to happen? \B matches the empty string in every other language I can think of.

Online reproducer: https://godbolt.org/z/8q6fehss7

CPython versions tested on:

3.11, 3.12

Operating systems tested on:

Linux

Linked PRs

• gh-124130: Notes on empty string corner case of category \B #124133
• [3.13] gh-124130: Notes on empty string corner case of category \B (GH-124133) #124328
• [3.12] gh-124130: Notes on empty string corner case of category \B (GH-124133) #124329
• gh-124130: Increase test coverage for \b and \B in regular expressions #124330
• [3.13] gh-124130: Increase test coverage for \b and \B in regular expressions (GH-124330) #124413
• [3.12] gh-124130: Increase test coverage for \b and \B in regular expressions (GH-124330) #124414
• gh-124130: Fix a bug in matching regular expression \B in empty string #127007

[brianschubert]

The current behavior (\B not matching "") appears to be intentional:

cpython/Lib/test/test_re.py

Lines 896 to 898 in aba42c0

	# However, an empty string contains no word boundaries, and also no
	# non-boundaries.
	self.assertIsNone(re.search(r"\B", ""))

Strictly speaking, re is working as designed here, even if that design is arguably "incorrect" with respect to other regex implementations.

Changing this behavior would technically be a feature request (not a bug), and would likely require a deprecation period. I'd recommend opening a thread on https://discuss.python.org/c/ideas/6 to see if there's community interest in changing this.

For reference, JavaScript treats the empty string as containing a non-word boundary:

>> /\b/.test("")
false
>> /\B/.test("")
true

[y5c4l3]

Another Perl reference:

$ perl -E 'say "Boundary" if "" =~ /\b/; say "Non-boundary" if "" =~ /\B/;'
Non-boundary

Seems that it is taking Perl as a baseline to some extent, as indicated in test_re:

cpython/Lib/test/test_re.py

Lines 896 to 901 in 9017b95

	# However, an empty string contains no word boundaries, and also no
	# non-boundaries.
	self.assertIsNone(re.search(r"\B", ""))
	# This one is questionable and different from the perlre behaviour,
	# but describes current behavior.
	self.assertIsNone(re.search(r"\b", ""))

However the comments really confuse me since the behavior for \b atm is same as Perl's but it's still claiming different from the perlre behaviour. Or have I missed some changes on the behavior of \b later on? If that was the case, I think it's also reasonable to align the behavior of \B to Perl's.

[terryjreedy]

However the comments really confuse me

I think both comments apply to the \B case, as that is the one different from Perl. Splitting the comment that way is a bit confusing if one does not know Perl, which most do not now. @serhiy-storchaka can comment, but I don't think we can really change this now.

[serhiy-storchaka]

This test was added in 5a045b9 (bpo-10713/gh-54922). It was not an assertion for the intended behavior, it was added to ensure that the current behavior would not change. Strictly speaking, the current behavior contradicts the documentation that says that \B is the opposite of \b.

Of course, it may be that the documentation is wrong. But taking into account that the current behavior differs from the behavior of many (if not all) other RE engines, that the code is most likely a copying error (it was not properly tested), that we already did several breaking changes related to zero-width matches in the past, and that it affects only very specific cases, I think that we can and should change this behavior.

It is preferable to emit a FutureWarning first, but I do not know how difficult to do this without producing false positives. If it is too difficult or impossible, we will have no other way as to change the behavior without warning.

[gpshead]

We've been reluctant to make any changes to the re module behavior in the past as people's existing code depends on all of its behaviors per Hyrum's Law.

[vstinner]

The best that we can do is to document the bug, if there is a bug, and explain that it's kept for backward compatibility.

[ZeroIntensity]

The linked PR is now just a documentation note.