Clang 14 rejects certain Unicode characters in identifiers that are accepted by Clang 13 and the C++ Standard

[tttapa]

Some Unicode characters like ₊ (U+208A) and other subscripts are rejected by Clang 14. These characters are in the allowed ranges for identifiers in the [lex.name] section of the C++ Standard. Recent versions of GCC and older versions of Clang do not raise any errors.

For example:

double foo(double xₖ, double xₖ₊₁) {
  return xₖ₊₁ - xₖ;
}

$ clang++-14 -c unicode.cpp -std=c++20                                                                                                                                                 
unicode.cpp:1:36: error: character <U+208A> not allowed in an identifier
double foo(double xₖ, double xₖ₊₁) {
                               ^
unicode.cpp:1:39: error: character <U+2081> not allowed in an identifier
double foo(double xₖ, double xₖ₊₁) {
                                ^
unicode.cpp:2:14: error: character <U+208A> not allowed in an identifier
  return xₖ₊₁ - xₖ;
           ^
unicode.cpp:2:17: error: character <U+2081> not allowed in an identifier
  return xₖ₊₁ - xₖ;
            ^
4 errors generated.

$ clang++-14 --version
Ubuntu clang version 14.0.1-++20220402053234+23d08271a4b2-1~exp1~20220402053315.111
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Is this a deliberate change or a regression bug from Clang 13 to 14?

[llvmbot]

@llvm/issue-subscribers-clang-frontend

[cor3ntin]

This is a deliberate change.
Clang 14 implemented https://wg21.link/p1949 which was standardized for all C++ versions and limits the set of allowed codepoints in identifiers to what is recommended by Unicode.

I do wonder if we could make that clearer in the diagnostic though

[tttapa]

Thanks for the quick reply!

That's unfortunate, these subscripts were really handy to use the mathematical notation from papers, formulas and pseudocode in the C++ implementation. Is there an option to get back the old behavior? (I didn't find one in https://clang.llvm.org/docs/ClangCommandLineReference.html but I might be searching for the wrong keywords.)

[intractabilis]

For some inexplicable reason ∂, 𝜕 are now not allowed. Dear Steve Downey, Zach Laine, Tom Honermann, Peter Bindels, and Jens Maurer why do you hate derivatives so much?

Dear Clang, please, don't adopt P1949R7 because it is ridiculous, unnecessary, and breaks existing code.

[intractabilis]

I posted a comment at the corresponded review page: https://reviews.llvm.org/D104975#3486313

[AaronBallman]

I agree that this behavior is intentional and some amount of broken code is expected as a result. I'm sorry you've been caught by that!

Is there an option to get back the old behavior?

There is not. We could perhaps elect to not implement this paper in older language modes (so it only happens in -std=c++2b and later) and we could elect to add a feature flag so you can opt into a non-conforming mode in C++23 and later. However, such a change is somewhat risky and something I'd like to avoid unless we see significant code breakage in the wild (system headers, major third-party library headers, a ton of individual user projects, etc). Previously, neither the C nor the C++ committee had a principled reason for what was or wasn't a valid character in an identifier when it came to Unicode characters. This caused real problems (including a high-score CVE in the same space) and so the committees both decided to defer to the Unicode consortium as to what is and isn't a valid character for an identifier (with one exception for _ as the low-line character was necessary for backwards compatibility). We do not wish to go back to ad hoc allowance/rejection of characters for identifiers, so we're most likely going to wait for the Unicode consortium to change their rules or the C or C++ committees to add exceptions.

That said, the fact that this code was broken and it's causing you pain is helpful for the standards bodies to understand the impact of the changes. We'll make sure this information is fed back to the standards bodies (it's already generated some discussion from the original report). And if it starts to look like more people are getting caught by this, we'll certainly consider what changes we can make to ease the burdens.

Dear Clang, please, don't adopt P1949R7 because it is ridiculous, unnecessary, and breaks existing code.

While you might be frustrated by the situation, please do not disparage the hard work of others as being ridiculous or unnecessary, and please follow our Code of Conduct: https://llvm.org/docs/CodeOfConduct.html

[intractabilis]

I insist that excluding math symbols like partial derivative is in my opinion unnecessary and strange. I would argue that 𝜕 has more sense than supporting emoji in identifiers. I am really sorry and regret that hard work of others was directed to something unnecessary and arbitrary, but hard work of others hardly can be a reason to change my opinion.

GCC supports 𝜕, and I hope it's not going to change.

[intractabilis]

I guess the reason why 𝜕 was excluded is somebody confused a mathematical operator, which is a function that acts on other functions or on some structured objects, with a programming language operator, which is professional lingo for a mathematical operation, or for some other language-specific weird operation. 𝜕 is a mathematical operator, for any purpose of a programmer it's just a letter.

However, the existence of an explanation doesn't make this decision good, logical, sane, or necessary. 𝜕 is much more useful than emoji. If I implement an algorithm from a paper, and the paper says 𝜕Ω, this is what would be absolutely natural to use in the code.

There is neither benefit nor sense in removing 𝜕.

[FrankHB]

I guess the reason why 𝜕 was excluded is somebody confused a mathematical operator, which is a function that acts on other functions or on some structured objects, with a programming language operator, which is professional lingo for a mathematical operation, or for some other language-specific weird operation. 𝜕 is a mathematical operator, for any purpose of a programmer it's just a letter.

I'd like to add some notes.

Generally speaking, a mathematical operator is an operator in the sense of programming languages, which derive the rules from math systems.

The design to make operators always in specific syntactic categories is language-specific. For example, operators are punctuations (rather than identifiers) in C and C++, while many Lisp dialects just treat operators same to elements in the head subform of a function application (i.e. operators = functions being applied to) and such operators can be named by identifiers. (To be accurate, punctuators here are the "operator-or-punctuator" category in C++; operators and puctuators will be in different categories later.)

I don't mean to change C/C++, but I'd argue the latter ("the Lisp style") is better than the former ("the ALGOL style") in contexts of language-agnostic meaning of the notion "operator". (I use "ALGOL" to suggest whether an operator can be defined be user is irrelavant here.)

Traditionally, math systems do not distinguish syntactic and semantic forms of elements like operators, as they are always self-evaluating. That is, thay will not change to anything else during the deduction in the system, unless combined with somthing other (the operands). This is OK in traditional uses (just interested in getting the results of some computations), but formally probamatic when you want to describe the underlying system in more detailed ways (say, operational semantics). To describe the precise behavior of the system, you have to differentiate whether an element (not necessarily a self-evaluating one) is evaluated (more formally, reduced to the normal form) or not, and mixing different contextual meanings of such elements would be a mess.

Languages like C and C++ are not totally formally defined. However, they still imply rules like deduction systems in their formal grammars. Specifically, C and C++ have notion of phases of translations, so lexical identical elements (even "self-evaluating" during the translation) can be actually different: identifiers of preprocessing tokens are not same to identifiers of tokens.

This example is important because it is exactly concerned in the meaning of "identifiers" handled here. I don't think it is the Unicode consortium's work to clarify the mapping from the meaning in specific programming languages to the definition in the Unicode specification. In particular, not all programming languages need the distinction. (C and C++ need it, because some of identifiers of preprocessing tokens would be converted to keywords of tokens instead of identifiers of tokens.) So, it would hardly come true "to establish conventions that will be followed by most/all programming languages" without further efforts with more careful analyses (which are closer than "the Lisp style" rather than "the ALGOL style").

The syntactic element 𝜕 is traditionally used to construct some specific kind of operators in the calculus. A single letter 𝜕 has no specific formal semantics without combining to other syntactic elements. Nevertheless, in more modern views, it can be treated as a higher-order constructor of the operators (to combine other symbols of variables together). This is essentially the same role of the lexeme lambda (or λ) in Lisp dialects: a single lambda names no special entity because special forms in Lisp do not reduce to Lisp values, only well-formed lambda forms are meaningful. So, lambda in such languages will not name a thing by user.

However, in some extended calculi, lambda can actually be derived as first-class objects. So, I suggest the fact that whether such an syntactic element can be solely used in well-fomed programs a language-specific detail, and language-agnostic treatment should allow such elements being plain variables in general (as λ not necessarily represents the constructor of lambda abstractions). To minimize the surprise, assigning the same lexical category (compared to other identifiers naming variables) to such elements seem necessary, whatever it would be an element of identifiers in the Unicode side.

[intractabilis]

Generally speaking, a mathematical operator is an operator in the sense of programming languages, which derive the rules from math systems.

It is not. What we call a math operator in a programming language, in real math, is called an operation. Never in my life I heard anyone calling addition a "plus operator" in a calculus class. Wikipedia even has two separate articles for math operators and for programming language operators.

I don't think it is the Unicode consortium's work to clarify the mapping from the meaning in specific programming languages to the definition in the Unicode specification.

Yes, I agree. I think that P1949 helps nobody.

The syntactic element 𝜕 is traditionally used to construct some specific kind of operators in the calculus.

Not necessarily. It's common to denote a boundary of Ω as 𝜕Ω. With some effort, I guess you can develop a theory where 𝜕 in this case will be an operator, but in most papers it's just a syntax sugar to denote a boundary.

[tttapa]

@AaronBallman, thank you for the detailed reply.

While I agree that serious issues and security vulnerabilities should be addressed and fixed retroactively for older standards, I feel this change goes way beyond that, for two reasons:

1. It potentially breaks a lot of working code, including e.g. code in papers or other publications, which will now forever be broken, even when explicitly compiling for older language standards.
2. There is a big difference between disallowing risky control characters because of security issues and disallowing normal, well-behaved mathematical characters and subscripts.

I'm in favor of ironing out some of the inconsistencies in the ranges of allowed characters, and addressing things like normalization may be important, but in my opinion, by suddenly disallowing perfectly reasonable characters, the current implementation of P1949 creates many more problems than it actually solves.

---

Aside from the issue of backwards compatibility, I'd like to motivate the use of certain Unicode characters:

I'd argue that it is perfectly reasonable to use variable names such as 𝜕Ω, as this 1. improves readability of the code, and 2. allows you to closely follow the notation used in the field (e.g. mathematical papers).

As an example from the code I'm currently working on:

// Compute forward-backward envelope
φₖ₊₁ = ψₖ₊₁ + 1 / (2 * γₖ₊₁) * pₖ₊₁ᵀpₖ₊₁ + grad_ψₖ₊₁ᵀpₖ₊₁;

With limited Unicode support, I have to write something like this:

// Compute forward-backward envelope
φ_k_plus_1 = ψ_k_plus_1 + 1 / (2 * γ_k_plus_1) * p_k_plus_1ᵀp_k_plus_1
           + grad_ψ_k_plus_1ᵀp_k_plus_1;

What could previously be effortlessly parsed and easily matched to the formula in the paper has now become an unreadable mess of letters and underscores.

In the first expression, the variable names are distinct and concise, and can be recognized at a glance. As a result, you can easily focus on the operations that are actually carried out on them. Thanks to the subscripts, you automatically focus on the actual names rather than on the k+1, which is less important.

In the second expression, variable names share the _k_plus_1 part and are harder to tell apart at a glance. Furthermore, because of the long variable names, the operators are farther apart, making it much harder to parse the expression as a whole.

Searching the code base of my current project, I found over a thousand matches for the subscripts 0 through 9.
Personally, I will not be refactoring the code, because 1. it is an unnecessary change to code that works perfectly fine as is, and 2. it would seriously hurt readability.
That unfortunately means I've had to downgrade Clang to version 13.

---

I noticed that P1949R7 states (https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2021/p1949r7.html#what-will-this-proposal-not-change):

5 What will this proposal not change?

5.1 The validity of “extended”" characters in identifiers

All current compilers allow characters outside the basic source character set directly in source today.

Does this leave room for Clang and other compilers to still allow these characters even though they might be outside of XID_*?

We'll make sure this information is fed back to the standards bodies

I appreciate that, thank you.
I realize that this is not an issue for the majority of programmers, but this is something that is quite important when dealing with mathematical expressions. For example, in Julia, another language that's often used in our field, φₖ₊₁, grad_ψₖ₊₁ᵀpₖ₊₁ and 𝜕Ω are all valid identifiers.

[intractabilis]

@tttapa I totally agree with everything you said. Btw, GCC has also implemented P1949 in version 12, but it still allows math symbols like 𝜕Ω and φₖ₊₁.

Here is a demonstration with a snapshot of GCC from the release 12 branch:

#include <iostream>

int main(int argc, char* argv[]) {
    auto 𝜕Ω = 4;
    auto φₖ₊₁ = 5;
    std::cout << "𝜕Ω = " << 𝜕Ω << std::endl;
    std::cout << "φₖ₊₁ = " << φₖ₊₁ << std::endl;
}

$ g++ --version
g++ (GCC) 12.0.1 20220504 (prerelease)
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ g++ -std=c++23 -o test test.cpp
$ ./test
𝜕Ω = 4
φₖ₊₁ = 5

Btw, before you, I didn't know there is Unicode for subscripts and superscripts. Now I am totally going to use it.