Skip to content

[HTML] Update comment highlighting to match WHATWG HTML standard #232

Closed
Closed
[HTML] Update comment highlighting to match WHATWG HTML standard#232
Labels
htmlCaused by the hypertext lexer

Description

@zufuliu
zufuliu
opened on Apr 1, 2024

See python/cpython#102555, for following snippet, all text before number is treated as comment by browser:

<!---->1 normal comment
<!--<!--->2 nested-comment
<!-->3 abrupt-closing-of-empty-comment
<!--->4 abrupt-closing-of-empty-comment
<!----!>5 incorrectly-closed-comment

currently only line 1 and line 2 are correctly handled.

https://html.spec.whatwg.org/multipage/parsing.html#parse-error-nested-comment

This error occurs if the parser encounters a nested comment (e.g., <!-- <!-- nested --> -->). Such a comment will be closed by the first occurring "-->" code point sequence and everything that follows will be treated as markup.

https://html.spec.whatwg.org/multipage/parsing.html#parse-error-abrupt-closing-of-empty-comment

This error occurs if the parser encounters an empty comment that is abruptly closed by a U+003E (>) code point (i.e., <!--> or <!--->). The parser behaves as if the comment is closed correctly.

https://html.spec.whatwg.org/multipage/parsing.html#parse-error-incorrectly-closed-comment

This error occurs if the parser encounters a comment that is closed by the "--!>" code point sequence. The parser treats such comments as if they are correctly closed by the "-->" code point sequence.

Patch html-comment-0401.zip

diff --git a/lexers/LexHTML.cxx b/lexers/LexHTML.cxx
index 0ed9f90b..942c6a56 100644
--- a/lexers/LexHTML.cxx
+++ b/lexers/LexHTML.cxx
@@ -1572,6 +1572,13 @@ void SCI_METHOD LexerHTML::Lex(Sci_PositionU startPos, Sci_Position length, int
 				state = SCE_H_COMMENT; // wait for a pending command
 				styler.ColourTo(i + 2, SCE_H_COMMENT);
 				i += 2; // follow styling after the --
+				chNext = SafeGetUnsignedCharAt(styler, i + 1);
+				if ((chNext == '>') || (chNext == '-' && SafeGetUnsignedCharAt(styler, i + 2) == '>')) {
+					// https://html.spec.whatwg.org/multipage/parsing.html#parse-error-abrupt-closing-of-empty-comment
+					i -= (ch == '>') ? 2 : 1;
+					chPrev = '-';
+					ch = '-';
+				}
 			} else if (isWordCdata(i + 1, i + 7, styler)) {
 				state = SCE_H_CDATA;
 			} else {
@@ -1843,7 +1850,11 @@ void SCI_METHOD LexerHTML::Lex(Sci_PositionU startPos, Sci_Position length, int
 			}
 			break;
 		case SCE_H_COMMENT:
-			if ((scriptLanguage != eScriptComment) && (chPrev2 == '-') && (chPrev == '-') && (ch == '>')) {
+			if ((scriptLanguage != eScriptComment) && (chPrev2 == '-') && (chPrev == '-') && (ch == '>' || (ch == '!' && chNext == '>'))) {
+				// https://html.spec.whatwg.org/multipage/parsing.html#parse-error-incorrectly-closed-comment
+				if (ch == '!') {
+					i += 1;
+				}
 				styler.ColourTo(i, StateToPrint);
 				state = SCE_H_DEFAULT;
 				levelCurrent--;

More tests may needed as there are restrictions listed at https://html.spec.whatwg.org/multipage/syntax.html#comments

Comments must have the following format:

The string "<!--".
Optionally, text, with the additional restriction that the text must not start with the string ">", nor start with the string "->", nor contain the strings "<!--", "-->", or "--!>", nor end with the string "<!-".
The string "-->".

Activity

  1. zufuliu
    added a commit that references this issue on Apr 1, 2024

    Update comment highlighting to match WHATWG HTML standard

    3196807

  2. zufuliu commented on Apr 1, 2024

    @zufuliu
    zufuliu
    on Apr 1, 2024
    ContributorAuthor

    Minor update previous patch:
    html-comment-0401-2.zip

    - i -= (ch == '>') ? 2 : 1;
+ if (chNext == '-') {
+ 	i += 1;
+ }
    diff --git a/lexers/LexHTML.cxx b/lexers/LexHTML.cxx
index 0ed9f90b..3d3e08e4 100644
--- a/lexers/LexHTML.cxx
+++ b/lexers/LexHTML.cxx
@@ -1572,6 +1572,15 @@ void SCI_METHOD LexerHTML::Lex(Sci_PositionU startPos, Sci_Position length, int
 				state = SCE_H_COMMENT; // wait for a pending command
 				styler.ColourTo(i + 2, SCE_H_COMMENT);
 				i += 2; // follow styling after the --
+				chNext = SafeGetUnsignedCharAt(styler, i + 1);
+				if ((chNext == '>') || (chNext == '-' && SafeGetUnsignedCharAt(styler, i + 2) == '>')) {
+					// https://html.spec.whatwg.org/multipage/parsing.html#parse-error-abrupt-closing-of-empty-comment
+					if (chNext == '-') {
+						i += 1;
+					}
+					chPrev = '-';
+					ch = '-';
+				}
 			} else if (isWordCdata(i + 1, i + 7, styler)) {
 				state = SCE_H_CDATA;
 			} else {
@@ -1843,7 +1852,11 @@ void SCI_METHOD LexerHTML::Lex(Sci_PositionU startPos, Sci_Position length, int
 			}
 			break;
 		case SCE_H_COMMENT:
-			if ((scriptLanguage != eScriptComment) && (chPrev2 == '-') && (chPrev == '-') && (ch == '>')) {
+			if ((scriptLanguage != eScriptComment) && (chPrev2 == '-') && (chPrev == '-') && (ch == '>' || (ch == '!' && chNext == '>'))) {
+				// https://html.spec.whatwg.org/multipage/parsing.html#parse-error-incorrectly-closed-comment
+				if (ch == '!') {
+					i += 1;
+				}
 				styler.ColourTo(i, StateToPrint);
 				state = SCE_H_DEFAULT;
 				levelCurrent--;

  3. zufuliu commented on Apr 1, 2024

    @zufuliu
    zufuliu
    on Apr 1, 2024
    ContributorAuthor

    The patch is still not ready as XML comment parsing may not follow the same rules.

  4. nyamatongwe
    added
    htmlCaused by the hypertext lexer
    on Apr 1, 2024

  5. nyamatongwe commented on Apr 1, 2024

    @nyamatongwe
    nyamatongwe
    on Apr 1, 2024
    Member

    Patch also needs an example file to ensure it stays fixed.

  6. zufuliu commented on Apr 2, 2024

    @zufuliu
    zufuliu
    on Apr 2, 2024
    ContributorAuthor

    html-comment-0402.zip

    diff --git a/lexers/LexHTML.cxx b/lexers/LexHTML.cxx
index 0ed9f90b..c39ed0e4 100644
--- a/lexers/LexHTML.cxx
+++ b/lexers/LexHTML.cxx
@@ -1572,6 +1572,18 @@ void SCI_METHOD LexerHTML::Lex(Sci_PositionU startPos, Sci_Position length, int
 				state = SCE_H_COMMENT; // wait for a pending command
 				styler.ColourTo(i + 2, SCE_H_COMMENT);
 				i += 2; // follow styling after the --
+				if (!isXml) {
+					// handle empty comment: <!-->, <!--->
+					// https://html.spec.whatwg.org/multipage/parsing.html#parse-error-abrupt-closing-of-empty-comment
+					chNext = SafeGetUnsignedCharAt(styler, i + 1);
+					if ((chNext == '>') || (chNext == '-' && SafeGetUnsignedCharAt(styler, i + 2) == '>')) {
+						if (chNext == '-') {
+							i += 1;
+						}
+						chPrev = '-';
+						ch = '-';
+					}
+				}
 			} else if (isWordCdata(i + 1, i + 7, styler)) {
 				state = SCE_H_CDATA;
 			} else {
@@ -1843,7 +1855,12 @@ void SCI_METHOD LexerHTML::Lex(Sci_PositionU startPos, Sci_Position length, int
 			}
 			break;
 		case SCE_H_COMMENT:
-			if ((scriptLanguage != eScriptComment) && (chPrev2 == '-') && (chPrev == '-') && (ch == '>')) {
+			if ((scriptLanguage != eScriptComment) && (chPrev2 == '-') && (chPrev == '-') && (ch == '>' || (!isXml && ch == '!' && chNext == '>'))) {
+				// close HTML comment with --!>
+				// https://html.spec.whatwg.org/multipage/parsing.html#parse-error-incorrectly-closed-comment
+				if (ch == '!') {
+					i += 1;
+				}
 				styler.ColourTo(i, StateToPrint);
 				state = SCE_H_DEFAULT;
 				levelCurrent--;

    test case (added <p> after comment to test potential off-by-one error and make the result in web browser readable):

    <!----><p>1 normal comment</p>
<!-- > and <!--><p>2 valid comment</p>
<!--><p>3 abrupt-closing-of-empty-comment</p>
<!---><p>4 abrupt-closing-of-empty-comment</p>
<!----!><p>5 incorrectly-closed-comment</p>
<!--!> <h1 value="--!><p>6 incorrectly-closed-comment</p>
<!--<!---><p>7 nested-comment</p>
<!--<!---!><p>8 nested-comment</p>

    Updated patch to only handle these special comments in HTML as XML has strict comment parsing descripted at
    https://www.w3.org/TR/xml/#sec-comments

    For compatibility, the string " -- " (double-hyphen) must not occur within comments.
    Note that the grammar does not allow a comment ending in --->.

    Firefox reports "unclosed token" for following:

    <?xml version="1.0" encoding="utf-8"?>
<root>
<!-->
</root>

<?xml version="1.0" encoding="utf-8"?>
<root>
<!--->
</root>

    and "not well-formed" for following:

    <?xml version="1.0" encoding="utf-8"?>
<root>
<!--<!--->
</root>

<?xml version="1.0" encoding="utf-8"?>
<root>
<!----!>
</root>

<?xml version="1.0" encoding="utf-8"?>
<root>
<!--<!---!>
</root>

    The XML comment double-hyphen problem could be kept as is or handled in another issue.

  7. zufuliu
    added a commit that references this issue on Apr 2, 2024

    Don't apply HTML special comment handling to XML.

    f9e2d3f
  8. nyamatongwe
    added a commit that references this issue on Apr 5, 2024

    #232 Match standard handling of comments.

    5dad19b
  9. nyamatongwe
    added
    committedIssue fixed in repository but not in release
    on Apr 5, 2024
  10. nyamatongwe
    closed this as completedon Apr 23, 2024
  11. rdipardo
    added a commit that references this issue on Aug 27, 2024

    Implement substyles for identifiers in HTML SCE_H_TAG, SCE_H_ATTRIBUT…

    1c6e3e5
  12. nyamatongwe
    removed
    committedIssue fixed in repository but not in release
    on Feb 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    htmlCaused by the hypertext lexer

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions