Do NFKC normalization in lexer

[steveklabnik]

Issue by catamorphism
Thursday Apr 19, 2012 at 22:13 GMT

For earlier discussion, see rust-lang/rust#2253

This issue was labelled with: A-frontend, A-unicode, I-crash, I-enhancement, P-low in the Rust repository

---

next_token_inner in the lexer has a comment saying to do NKFC normalization. I have no idea what that is, but I guess we should do it.

reference: NFKC is one of four unicode Normalization Forms.

reference: UAX-31 supplies guidelines for use of normalization with identifiers.

[m13253]

I support NFC instead of NFKC.

Because I don't think we should normalize "⒛" into "20.".

[Centril]

Closing in favor of #2457.