Repository navigation
CWG3187 [lex.phases] Replacement of universal-character-names in translation phase three #893
Description
Activity
Why is it a CWG issue if compilers are buggy? Please post bug reports to the respective implementation.
Since there is no preprocessing token of the form \u03C0 this is invalid, even though the intent is to allow it.
Why do you believe the "intent" is to allow this?
Why is it a CWG issue if compilers are buggy? Please post bug reports to the respective implementation.
P2314 says "However, it turns out that all major implementations already implement what this paper specifies, i.e. no implementation provides an escaped UCN.", so I assumed the specification is wrong given that existing compilers do not implement it. Moreover, it creates quite a large discrepancy with C.
Since there is no preprocessing token of the form \u03C0 this is invalid, even though the intent is to allow it.
Why do you believe the "intent" is to allow this?
That is what P2621 says "We propose that spliced UCNs be supported because, in addition to 3/4 of surveyed compilers supporting it, it falls off naturally of the specification: splicing happens before any other form of tokenization and supporting it avoid special-casing this oddity.".
The "splicing" that P2621 talks about is line splicing, which is in phase 2 and does work with the current specification.
However,
##(concatenation) in the current specification cannot produce UCNs, contrary to the rather broad stroke in P2621 in the note that it can. Given that the note is not supported by normative text, it's probably best to strike it, or invert its statement.Is your larger example with
__VA_ARGS__inconsistent with the claim in P2314 that all major implementations already do the right thing with the example shown there?The "splicing" that P2621 talks about is line splicing, which is in phase 2 and does work with the current specification.
Looks like I misunderstood that, but P2621 clearly assumes that what it did was sufficient to make concatenation forming universal character names allowed.
Is your larger example with
__VA_ARGS__inconsistent with the claim in P2314 that all major implementations already do the right thing with the example shown there?#define S(...)#__VA_OPT__(__VA_ARGS__) #define X \u03C0 S(S(X))
The current wording says that
\u03C0should be replaced byπhere during tokenization, soS(X)should be"π"andS(S(X))should be"\"π\"". However, existing implementations do not do this soS(X)becomes"\u03C0"andS(S(X))becomes"\"\\u03C0\""(in C it could be"\"\u03C0\""as well). The differences between"π"and"\u03C0"are only observable when stringizing again which will reveal that their spelling is different. The right side of the table is correct when not considering stringizing the results. Though I do not think the left side of the table is correct,S(K\u00f6ppe)would become"K\u00f6ppe"not"K\\u00f6ppe"in C++20.Here is another scenario where implementations do not implement the current wording:
#define H <\u03C0>//not parsing a header name, just three tokens #include H #include<π>//should include the same file twice, but implementations do not do that
The "splicing" that P2621 talks about is line splicing, which is in phase 2 and does work with the current specification.
P2621 explicitly lists both line splicing and concatenation as motivations:
#define CONCAT(x, y) x ## y int CONCAT(\, u0393) = 0; // UB: universal character name formed by macro expansion
Given this and the fact that clang, gcc, and even msvc all support UCNs formed through concatenation, I think that the proposal intended to allow both spliced UCNs and UCNs produced by
##.However,
##(concatenation) in the current specification cannot produce UCNs, contrary to the rather broad stroke in P2621 in the note that it can. Given that the note is not supported by normative text, it's probably best to strike it, or invert its statement.While it could be made clearer, I don't think that the note conflicts with the normative text. The preprocessor must immediately determine if the result of the
##operator forms a valid preprocessing-token, essentially repeating the process that initially occurs in Phase 3. One could argue that the "preprocessing token recognition" was intended to include universal-character-name replacement. I don't think that making this explicit in the definition of the##operator would be a huge issue.Agreed. Phase 3 is the only place where we form preprocessing tokens (and check their validity), so assuming that
##feeds into phase 3 (where UCNs are formed and replaced, too) seems reasonable.While it could be made clearer, I don't think that the note conflicts with the normative text. The preprocessor must immediately determine if the result of the
##operator forms a valid preprocessing-token, essentially repeating the process that initially occurs in Phase 3. One could argue that the "preprocessing token recognition" was intended to include universal-character-name replacement. I don't think that making this explicit in the definition of the##operator would be a huge issue.If it is considered to go through phase three again then can comments be formed in concatenation?
#define C(X,Y)X##Y C(/,/)//OK?
Additionally, the wording that makes invalid concatenations invalid in [cpp.concat] "If the result is not a valid preprocessing token, the program is ill-formed." would not make sense as phase three never forms invalid preprocessing tokens, so presumably the following would also be OK:
int C(*,*)p;//same C as before, phase three would just split these into two * tokens
So I do not see how the current wording implies that it undergoes phase three again.
Hm...
"phase three never forms invalid preprocessing tokens"
Phase 3 is [lex.pptoken], and e.g. the last sentence of p1 there talks about "ill-formed". However, it feels self-contradictory to talk about an "invalid preprocessing token": Either something is a preprocessing token, or it doesn't match the lexer grammar for preprocessing-token, and then it's simply not a preprocessing token (not even an invalid one).
If it is considered to go through phase three again then can comments be formed in concatenation?
I can't think of any cases where the answer to this question would matter, can you? If concatenation can form comments, then the result (a partial comment) is not a preprocessing token, and the program is ill-formed. If concatenation cannot form comments, then the result (
//) is not a valid preprocessing token, and the program is ill-formed.Additionally, the wording that makes invalid concatenations invalid in [cpp.concat] "If the result is not a valid preprocessing token, the program is ill-formed." would not make sense as phase three never forms invalid preprocessing tokens
I don't think the meaning of that sentence actually changes if you remove the word "valid". The point is that the result must be exactly one preprocessing token. I think "valid" is just intended to clarify that the
##operator can't just stuff arbitrary sequences of characters into a single instance of the last preprocessing-token category and call it a day.Phase 3 is [lex.pptoken], and e.g. the last sentence of p1 there talks about "ill-formed". However, it feels self-contradictory to talk about an "invalid preprocessing token": Either something is a preprocessing token, or it doesn't match the lexer grammar for preprocessing-token, and then it's simply not a preprocessing token (not even an invalid one).
It's not particularly obvious to me why that wording is there. Either the excluded characters could be added to the definition of the last preprocessing-token category in the grammar, or those cases could be handled along with the rest of the partial preprocessing tokens and partial comments are not so trivially accounted for.
I don't think the meaning of that sentence actually changes if you remove the word "valid". The point is that the result must be exactly one preprocessing token. I think "valid" is just intended to clarify that the
##operator can't just stuff arbitrary sequences of characters into a single instance of the last preprocessing-token category and call it a day.The way I interpret it is "(not a valid) preprocessing token", rather than "not a (valid preprocessing token)".
The way I interpret it is "(not a valid) preprocessing token", rather than "not a (valid preprocessing token)".
I suppose that is technically ambiguous, but I think most people would use "an invalid preprocessing token" to convey the former meaning. I also think we can apply a bit of common sense to arrive at the conclusion that concatenating two preprocessing tokens was never intended to be able to produce two preprocessing tokens. Is that even concatenation?
I suppose that is technically ambiguous, but I think most people would use "an invalid preprocessing token" to convey the former meaning. I also think we can apply a bit of common sense to arrive at the conclusion that concatenating two preprocessing tokens was never intended to be able to produce two preprocessing tokens. Is that even concatenation?
It would be unambiguous if it was interpreted as not undergoing phase three again, as "the preceding preprocessing token is concatenated with the following preprocessing token" does not seem to imply that. If the intent was currently to repeat phase three, I would expect the wording to more clearly forbid multiple preprocessing tokens attempting to be formed. When concatenating two
*preprocessing tokens the result would not be a valid preprocessing token, as there is no token that has the representation**. So the current surrounding wording seems to align more with the idea that it does just a concatenation, rather than concatenating the spellings and doing phase three again.Thanks for the discussion. See CWG3187
- changed the title
[-][lex.phases] Replacement of universal-character-names in translation phase three[/-][+]CWG3187 [lex.phases] Replacement of universal-character-names in translation phase three[/+]on May 3, 2026 For MSVC specifically in the first example, it only seems to accept it with the old preprocessor. Adding /Zc:preprocessor will cause it to generate a warning.
I think the proposed partial resolution leaves an issue regarding the unspecified order of evaluation of multiple
##operators.For both object-like and function-like macro invocations, before the replacement list is reexamined for more macro names to replace, each instance of a ## preprocessing token in the replacement list (not from an argument), together with its immediately preceding and immediately following preprocessing token, is deleted and
the preceding preprocessing token is concatenated with the following preprocessing tokenreplaced by a single preprocessing token formed by concatenating the spellings of the preceding and following preprocessing tokens and replacing any universal-character-names with the characters they designate (5.3.1 [lex.charset]). If no single valid preprocessing token can be formed (5.5 [lex.pptoken]), the program is ill-formed. Placemarker preprocessing tokens are handled specially: concatenation of two placemarkers results in a single placemarker preprocessing token, and concatenation of a placemarker with a non-placemarker preprocessing token results in the non-placemarker preprocessing token.For example:
#define CONCAT(x, y, z) x ## y ## z CONCAT(\, u, 1234)
After argument substitution, the relevant replacement list is:
\ ## u ## 1234The order of evaluation of
##operators is unspecified. Under the proposed wording, this can affect whether the program is well-formed.If the left
##is processed first,\anduare concatenated, producing the spelling\u. This does not form a single valid preprocessing token, so the program is ill-formed.If the right
##is processed first,uand1234are concatenated to form the identifieru1234. The remaining concatenation then produces\u1234, which is recognized as a universal-character-name and replaced by the character it designates. This can therefore result in a valid preprocessing token.Thus, the same macro invocation can be either ill-formed or well-formed depending on the order in which the
##operators are evaluated, even though that order is specified as unspecified.I think the proposed resolution should clarify this interaction between multiple
##operators and the new UCN handling, in particular whether the validity of the concatenation is permitted to depend on the unspecified evaluation order.Thus, the same macro invocation can be either ill-formed or well-formed depending on the order in which the
##operators are evaluated, even though that order is specified as unspecified.That is not a new result of this wording, consider
CONCAT(<,=,>),CONCAT(u,8,""), orCONCAT(0X1,.,AP0).That is not a new result of this wording, consider
CONCAT(<,=,>),CONCAT(u,8,""), orCONCAT(0X1,.,AP0).Is there an issue about it? Current definition of "unspecified behavior" ([defns.unspecified]) presumes a well-formed program, but here a source file is unspecified whether it is well-formed or not.
That is not a new result of this wording, consider
CONCAT(<,=,>),CONCAT(u,8,""), orCONCAT(0X1,.,AP0).Is there an issue about it? Current definition of "unspecified behavior" ([defns.unspecified]) presumes a well-formed program, but here a source file is unspecified whether it is well-formed or not.
No issues for this that I know. And there are plenty of other ways to construct programs that are well-formed depending upon unspecified results. For one example:
#if 9223372036854775807+1 #endif
Is this valid or not? That depends upon the width of
std::intmax_t.
Full name of submitter (unless configured in github; will be published with the issue): Jay Ghiron
Reference (section label): [lex.phases]/1.3
Issue description:
Consider the following two macro definitions:
According to translation phase three, these macro definitions should be equivalent because the universal-character-name should be replaced by the character it represents (U+03C0). However, GCC and MSVC reject this and consider them to be different. Clang accepts them as the same, but does not actually treat them the same:
Additionally, the current wording forbids the following:
Since there is no preprocessing token of the form
\u03C0this is invalid, even though the intent is to allow it.Related: P2314 and P2621