Repository navigation
CWG3224 [cpp.replace.general] p14: Clarification of directive-like preprocessing tokens in macro arguments #975
Description
Activity
What's the behavior of existing implementations for these examples?
In any case, I don't think those two rules conflict, but they are complementary. [cpp.replace.general] p14 is talking about the argument list (before any replacement happens). [cpp.rescan] p4 is talking about the result of the replacement and the rescanning. At that stage, preprocessing directives are no longer recognized (and the token sequence survives until phase 6, where they are ill-formed, because
#is not a valid phase 7 token).In any case, I don't think those two rules conflict, but they are complementary. [cpp.replace.general] p14 is talking about the argument list (before any replacement happens). [cpp.rescan] p4 is talking about the result of the replacement and the rescanning. At that stage, preprocessing directives are no longer recognized (and the token sequence survives until phase 6, where they are ill-formed, because
#is not a valid phase 7 token).That's exactly my viewpoint. However even though the first
ECHOexample would be ill-formed afterward, the secondSTRexample works.What's the behavior of existing implementations for these examples?
See godbolt. It seems that their preprocessors remain to respect the old specification ([cpp.replace.general]/2 used to be a UB) so they accept both
STR(#py comment)andSTR(PY comment).Edit: adding C++ version flag (-std=c++26 etc.) doesn't result in any change.
the old specification ([cpp.replace.general]/2 used to be a UB)
Ah, we changed this in "P2843R3 Preprocessing is never undefined". My guess is implementations haven't caught up to this change. Feel free to post bug reports for
STR(#py comment) // ill-formedthe old specification ([cpp.replace.general]/2 used to be a UB)
Ah, we changed this in "P2843R3 Preprocessing is never undefined". My guess is implementations haven't caught up to this change. Feel free to post bug reports for
STR(#py comment) // ill-formedThat's why I post this issue: is
STR(PY comment)well-formed after P2843R2 instead of being an edge case of a UB? In the past, whetherSTR(PY comment)is well-formed depends on a UB so it doesn't matter at all.P2843 wanted to remove UB from the preprocessor.
Given the current state of the wording in the standard, is there any reason to believe that
STR(PY comment)would be ill-formed?P2843 wanted to remove UB from the preprocessor.
Given the current state of the wording in the standard, is there any reason to believe that
STR(PY comment)would be ill-formed?I think
STR(PY comment)is well-formed according to the current standard. I wonder if this case needs an informative note or example, since it used to associate with a UB and now the UB is removed but this case becomes well-formed.Why was it UB? [cpp.replace.general]/14 used to say "UB" (instead of ill-formed), but we agreed this rule is not triggered by your example.
I found another ambiguity here.
If there are sequences of preprocessing tokens within the list of arguments that would otherwise act as preprocessing directives, the program is ill-formed.
Does it mean that there shall be no
#within the list of arguments since a single#suffices to form a null directive?Or we should consult [cpp.pre]/2 and whitespaces matter?
A preprocessing directive consists of a sequence of preprocessing tokens that satisfies the following constraints: At the start of translation phase 4, the first preprocessing token in the sequence, referred to as a directive-introducing token , begins with the first character in the source file (optionally after whitespace containing no new-line characters) or follows whitespace containing at least one new-line character, and is
- a # preprocessing token, or
- ...
The last preprocessing token in the sequence is the first preprocessing token within the sequence that is immediately followed by whitespace containing a new-line character.
Consider:
STR( # ) // ill-formed STR(#) // ill-formed? STR(# ) // ill-formed? STR( #) // ill-formed? STR(bluh # bluh) // ill-formed? STR(bluh#) // ill-formed? STR(bluh#if) // ill-formed? (triggered by #, not by incomplete #if) STR(bluh# bluh) // ill-formed? STR(bluh #if) // ill-formed?
Should I open a new issue? Or may I edit the current issue to update this problem?
Why was it UB? [cpp.replace.general]/14 used to say "UB" (instead of ill-formed), but we agreed this rule is not triggered by your example.
Yes if we adopt my interpretation, then
STR(PY comment)doesn't belong to the UB.I found another ambiguity here.
If there are sequences of preprocessing tokens within the list of arguments that would otherwise act as preprocessing directives, the program is ill-formed.
Does it mean that there shall be no
#within the list of arguments since a single#suffices to form a null directive?Or we should consult [cpp.pre]/2 and whitespaces matter?
Yes, the "otherwise" means if macro expansion did not take place I believe. So only the examples that have the
#on the start of a line are invalid.Yes, the "otherwise" means if macro expansion did not take place I believe. So only the examples that have the
#on the start of a line are invalid.How about the ending condition?
The last preprocessing token in the sequence is the first preprocessing token within the sequence that is immediately followed by whitespace containing a new-line character.
Do you mean that only
STR(bluh # bluh)is ill-formed? Or
STR(bluh #if)
is also ill-formed? Note that here
#ifis imcomplete and cannot act as a preprocessing directive otherwise.Yes, the "otherwise" means if macro expansion did not take place I believe. So only the examples that have the
#on the start of a line are invalid.How about the ending condition?
The last preprocessing token in the sequence is the first preprocessing token within the sequence that is immediately followed by whitespace containing a new-line character.
Do you mean that only
STR(bluh
#
bluh)
is ill-formed? OrSTR(bluh
#if)
is also ill-formed? Note that here#ifis imcomplete and cannot act as a preprocessing directive otherwise.With your previous example:
STR( # ) // ill-formed STR(#) // valid STR(# ) // valid STR( #) // ill-formed STR(bluh # bluh) // ill-formed STR(bluh#) // valid STR(bluh#if) // valid STR(bluh# bluh) // valid STR(bluh #if) // ill-formed
The
#if)should still be considered a preprocessor directive, just an invalid one. Note that it cannot be treated as a normal text line "A sequence of preprocessing tokens is only a text-line if it does not begin with a directive-introducing token." [cpp.pre]/3. Since it is not a valid if-group, it appears that the whole file is simply not a valid preprocessing-file like the example after the quoted text.As for that ending condition, I do not understand the point you are making. "sequences of preprocessing tokens within the list of arguments" does not imply that it has to be a whole argument, if that is what you were thinking of.
As for that ending condition, I do not understand the point you are making. "sequences of preprocessing tokens within the list of arguments" does not imply that it has to be a whole argument, if that is what you were thinking of.
Well [cpp.pre]/2 actually specifies two constraints on the "sequence of preprocessing tokens" that to be considered as a preprocessing directive:
- The first pp token shall be at the start of a line (optionally after some whitespace containing no new-line).
- The last pp token shall be immediately followed by some whitespace containing a new-line.
In my opinion, the second constraint is somewhat more important since otherwise there are two possible "sequences of preprocessing tokens" in
STR( #bluh)that would otherwise act as preprocessing directives (one is the null directive
#and another is the conditionally-supported directive#bluh). Afterall, there is no constraint on when to stop if the second constraint is omiited.My intention to use imcomplete
#ifis to distinguish whether the former case (#itself with immediately following pp tokens) is ill-formed. If the if-section causes complexity, one can useSTR( #line) // is the # itself forms a sequence of pp tokens that would otherwise act as pp-directive?
instead to prohibit the latter case (
#lineneeds more pp tokens) as a valid preprocessing directive otherwise. Don't you agree that "act as" may need the pp-directive to be valid? If we adopt this interpretation of "act as pp-directive", then#itself would make it ill-formed even though#linewould not. However this divergence may result in no difference: nevertheless it would be ill-formed because of the appearance of#(my opinion) or because of the invalid but sufficient appearance of#line(your opinion).As for that ending condition, I do not understand the point you are making. "sequences of preprocessing tokens within the list of arguments" does not imply that it has to be a whole argument, if that is what you were thinking of.
Well [cpp.pre]/2 actually specifies two constraints on the "sequence of preprocessing tokens" that to be considered as a preprocessing directive:
- The first pp token shall be at the start of a line (optionally after some whitespace containing no new-line).
- The last pp token shall be immediately followed by some whitespace containing a new-line.
In my opinion, the second constraint is somewhat more important since otherwise there are two possible "sequences of preprocessing tokens" in
STR(
#bluh)
that would otherwise act as preprocessing directives (one is the null directive#and another is the conditionally-supported directive#bluh). Afterall, there is no constraint on when to stop if the second constraint is omiited.Does it matter which directive it is considering? Though I think it should be considering the whole
#bluh), perhaps the wording should be adjusted to not imply that the whole directive needs to be in the arguments.My intention to use imcomplete
#ifis to distinguish whether the former case (#itself with immediately following pp tokens) is ill-formed. If the if-section causes complexity, one can useSTR(
#line) // is the # itself forms a sequence of pp tokens that would otherwise act as pp-directive?
instead to prohibit the latter case (#lineneeds more pp tokens) as a valid preprocessing directive otherwise. Don't you agree that "act as" may need the pp-directive to be valid? If we adopt this interpretation of "act as pp-directive", then#itself would make it ill-formed even though#linewould not. However this divergence may result in no difference: nevertheless it would be ill-formed because of the appearance of#(my opinion) or because of the invalid but sufficient appearance of#line(your opinion).I do not think there is any reason to outright forbid
#tokens entirely in preprocessing macros, that was surely never the intent. The intent as far as I understand is that if a macro invocation starts in a text line, it shall not cross over into lines other than text lines. For example:#define FOO(...) FOO( #include<iostream> )
This is not valid, but
FOO(#)has no issues with it being interpreted as a preprocessing directive so it does not make sense to forbid it. For macro invocations that start in lines other than text lines, there is also a constraint that they must be entirely contained within that line.So, we've established that the original examples are broken because the
#isn't at the start of a line (modulo whitespace).Do you still want an example for this:
STR( # ) // ill-formed(and yes, the point here is the
#symbol appearing before macro expansion)?I do not think there is any reason to outright forbid # tokens entirely in preprocessing macros, that was surely never the intent. The intent as far as I understand is that if a macro invocation starts in a text line, it shall not cross over into lines other than text lines. For example:
#define FOO(...)
FOO(
#include<iostream>
)
This is not valid, but FOO(#) has no issues with it being interpreted as a preprocessing directive so it does not make sense to forbid it. For macro invocations that start in lines other than text lines, there is also a constraint that they must be entirely contained within that line.I admit at the very beginning that this example (with both heading and trailing new-line) is ill-formed without doubt.
The detail
#bluh)of your response reminds me that maybe you interpret [cpp.replace.general]/14The sequence of preprocessing tokens bounded by the outside-most matching parentheses forms the list of arguments for the function-like macro...
as including the bounding parentheses. If so, then I can understand why you cannot tell the difference between the example showing above and the following
#define FOO(...) FOO( #include<iostream>)
since I interpret it as excluding the bounding parentheses. For this example
- either we include the parentheses (your opinion), realize
#include <iostream>)is an invalid pp-directive, and its subsequences#include <iostream>or#do not satisfy the ending constarint, - or we exclude the parentheses (my opinion), then there is no sequence of pp-tokens that satisfies the ending constraint because of the
)followed immediately.
Thus whether an invalid pp-directive serves as "otherwise act as a pp-directive" does matter; it would determine the well-formedness in the former case.
Thus whether the ending constaint needs to be satisfied does matter; it would determine the well-formedness in the latter case.
It's worth saying that even if we include the parentheses, consider:
#define FOO(...) FOO( #include<iostream>) FOO()
Then
#include<iostream>)violates the ending constraint and perhaps is no longer to be a pp-directive.However, I think the correct interpretation based on the context is exlcuding parentheses, since [cpp.replace.general]/14 continues as
...The individual arguments within the list are separated by comma preprocessing tokens, but comma preprocessing tokens between matching inner parentheses do not separate arguments.
If we including parentheses, then it seems that the individual arguments of
MACRO(a,b)would be(aandb), which is absurd (see also CWG 2003).Does it matter which directive it is considering? Though I think it should be considering the whole #bluh), perhaps the wording should be adjusted to not imply that the whole directive needs to be in the arguments.
Yes, the key point is whether the whole directive needs to be in the list of arguments. Since we'd better exlcude the bounding parentheses, if we moreover require the sequence of pp-tokens to satisfy the two constraints, then
MACRO(#) // well-formed MACRO(# ) // well-formed MACRO( #) // well-formed MACRO( # ) // ill-formed
which may be a satisfying resolution.
- either we include the parentheses (your opinion), realize
So, we've established that the original examples are broken because the
#isn't at the start of a line (modulo whitespace).Do you still want an example for this:
STR( # ) // ill-formed(and yes, the point here is the
#symbol appearing before macro expansion)?A detailed example such as
MACRO(#) // well-formed MACRO(# ) // well-formed MACRO( #) // well-formed MACRO( # ) // ill-formed #define HASH # MACRO( HASH ) // well-formed
might be helpful.
MACRO(
#) // well-formed#)is a conditionally-supported-directive.MACRO(
#) // well-formed#)is a conditionally-supported-directive.As I said above, we'd better exclude the bounding parentheses, so
If there are sequences of preprocessing tokens within the list of arguments that would otherwise act as preprocessing directives, the program is ill-formed.
means that
#)is NOT within the list of arguments which consists of merely a single#.#define FOO(...)
FOO(
#include)
since I interpret it as excluding the bounding parentheses. For this example- either we include the parentheses (your opinion), realize
#include <iostream>)is an invalid pp-directive, and its subsequences#include <iostream>or#do not satisfy the ending constarint, - or we exclude the parentheses (my opinion), then there is no sequence of pp-tokens that satisfies the ending constraint because of the
)followed immediately.
It is not really about specifically the closing
), I think the whole line should be interpreted as part of the directive. Consider the following:#define LPAREN ( #define FOO(...) #ifdef X FOO( #elif LPAREN 1) 42 #endif
Does this create an elif-group if and only if X is not defined? Or does it need to examine what FOO is even when X is not defined, therefore not actually ignoring the skipped tokens?
#)is a conditionally-supported-directive.As I said above, we'd better exclude the bounding parentheses, so
Grammatically it forms a conditionally-supported-directive, it cannot be a text-line and it cannot be another form of directive. I do not see how it can be understood as anything else. The grammar of preprocessing files does not depend upon macro expansion.
- either we include the parentheses (your opinion), realize
Grammatically it forms a conditionally-supported-directive, it cannot be a text-line and it cannot be another form of directive. I do not see how it can be understood as anything else. The grammar of preprocessing files does not depend upon macro expansion.
I think I get your point. So you opinion is
MACRO( #)is well-formed, since it would not be parsed as a multi-line macro invocation (which can only happen within text lines) so it has nothing to do with [cpp.replace.general]/14?
But if so, [cpp.replace.general]/14 seems to be redundant since there is actually no ill-formed case. Consider:
MACRO(#) // well-formed macro invocation MACRO(# ) // well-formed macro invocation MACRO( #) // well-formed conditionally-supported directive; no macro invocation here MACRO( # ) // well-formed null directive; no macro invocation here MACRO( #include <iostream>) // ill-formed #include directive; no macro invocation here
Is there any ill-formed case for which [cpp.replace.general]/14 is needed?
Furthermore, if this analysis does work, this UB was also never triggered before?
Grammatically it forms a conditionally-supported-directive, it cannot be a text-line and it cannot be another form of directive. I do not see how it can be understood as anything else. The grammar of preprocessing files does not depend upon macro expansion.
I think I get your point. So you opinion is
MACRO(
#)
is well-formed, since it would not be parsed as a multi-line macro invocation (which can only happen within text lines) so it has nothing to do with [cpp.replace.general]/14?The semantics of
#)are entirely up to the implementation, so I would say it is conditionally well-formed (that is how I understand conditionally supported) and can do absolutely anything. Programs intending to be strictly conforming cannot use it, even in skipped conditional inclusions (at least that is how Clang justifies supporting#elifdefin all language modes, when it would be a conditionally-supported-directive aka non-directive in earlier versions).But if so, [cpp.replace.general]/14 seems to be redundant since there is actually no ill-formed case. Consider:
MACRO(#) // well-formed macro invocation
MACRO(#
) // well-formed macro invocation
MACRO(
#) // well-formed conditionally-supported directive; no macro invocation here
MACRO(
#
) // well-formed null directive; no macro invocation hereThis is not valid:
#define F() F(
It would not make sense that adding a preprocessing directive then a closing parenthesis after would suppress macro expansion.
This is not valid:
#define F()
F(
It would not make sense that adding a preprocessing directive then a closing parenthesis after would suppress macro expansion.You are absolutely right. Now consider:
MACRO(#) // well-formed macro invocation MACRO(# ) // well-formed macro invocation MACRO( #) // ill-formed? MACRO( # ) // well-formed null directive; ill-formed macro invocation
The remaining problem is that whether the following example is well-formed or not
#define FOO(...) FOO( #line )
depends on which directive goes first. If
#linegoes first then#line )is invalid hence ill-formed. If#define FOO(...)goes first thenFOO(leads a list of arguments consisting of merely#line(excluding parentheses). But#lineitself does not form a preprocessing directive to trigger [cpp.replace.general]/14, while#line )does but is not within the list of arguments.Perhaps we should consider your suggestion on changing the wording of [cpp.replace.general]/14.
However, the if-sections coincidently work well. Consider:
#define LPAREN ( #define FOO(...) #ifdef X FOO( #elif LPAREN 1) (42 #endif )
To parse a preprocessing-file first, one needs to know when to relax syntax rules. Therefore evaluation of
#ifdef Xis sequenced before macro invocations associated with#define FOO(...). When the first#ifdef Xis met, we should see these directives above to determine whether the controling expression evaluates to zero. This split into two cases:- if
Xis not defined, then the if group is skipped and we try to recognize a strict#elifdirective. In this case theFOO(is skipped and#elif LPAREN 1)is expanded to be valid#elif ( 1). This branch works well and only(42)survives; - if
Xis defined, then the if group is valid and the following#elifcould have relaxed syntax. In this case theFOO(is not skipped and thus demands for a list of arguments. The following#elifdirective ending with)is skipped but#endifisn't ([cpp.cond]/13). So the result isFOO(\n#endif\n), which is ill-formed without doubt.
(If there is no
)after#endifandXis defined thenFOO(has no terminating), which is clearly ill-formed.)Also consider:
#define LPAREN ( #define FOO(...) FOO( #if LPAREN X) ) #else ) #endif
The file is processed to contain either
FOO(\n#if(1)\n)(ifXnonzero) orFOO(\n#else\n)(ifXzero) both ill-formed.- if
#define FOO(...)
FOO(
#line )depends on which directive goes first. If
#linegoes first then#line )is invalid hence ill-formed. If#define FOO(...)goes first thenFOO(leads a list of arguments consisting of merely#line(excluding parentheses). But#lineitself does not form a preprocessing directive to trigger [cpp.replace.general]/14, while#line )does but is not within the list of arguments.Perhaps we should consider your suggestion on changing the wording of [cpp.replace.general]/14.
If it is necessary to clearly forbid this, then I think it would be worth adding wording that the entire directive does not need to be contained within the arguments for this condition to occur. Though because this wording is from C (aside from the change to make it ill-formed instead of undefined) it would make sense to file an issue to WG14 too.
If it is necessary to clearly forbid this, then I think it would be worth adding wording that the entire directive does not need to be contained within the arguments for this condition to occur. Though because this wording is from C (aside from the change to make it ill-formed instead of undefined) it would make sense to file an issue to WG14 too.
I think there is a similar situation in [cpp.rescan]/1:
After all parameters in the replacement list have been substituted and # and ## processing has taken place, all placemarker preprocessing tokens are removed. Then the resulting preprocessing token sequence is rescanned, along with all subsequent preprocessing tokens of the source file, for more macro names to replace.
Suggested resolution:
Change [cpp.replace.general]/14:If there are sequences of preprocessing tokens within the list of arguments that along with all preprocessing tokens immediately followed would otherwise act as preprocessing directives, the program is ill-formed.
I have editted the original issue to reveal the discussion.
- changed the title
[-][cpp.replace.general] p14: Clarification of directive-like preprocessing tokens in macro arguments[/-][+]CWG3224 [cpp.replace.general] p14: Clarification of directive-like preprocessing tokens in macro arguments[/+]on Aug 23, 2026
Full name of submitter (unless configured in github; will be published with the issue): Yin Xinyu
Reference (section label): [cpp.replace.general]
Issue description:
[cpp.replace.general]/14 says:
The intention is possibly avoiding multi-line macro invocations that cross text lines and preprocessing directives ([cpp.pre]). Consider:
Although it's impossible to perform a multi-line macro invocation starting at a preprocessing directive, the current wording would accept the following
since the conditionally-supported directive
#bluh ) bluhis not within the list of arguments consisting of merely#bluh.Implementations have no consensus on how to deal with this case (godbolt). By the way they have not even diagnosed the first example as ill-formed.
This can result in an even weirder situation, although results are all ill-formed. Consider:
and
It's hard to tell what would happen, where
Xmay be in different branches.To avoid these complex and anti-intuitive circumstances, we'd better edit the wording of [cpp.replace.general]/14 to forbid these cases.
If an application such as stringization is needed, the following is well-formed.
The arguments in
STR2is safe, considering [cpp.rescan]/4Suggested resolution:
Change [cpp.replace.general]/14: