Skip to content

Fix tokenizer handling of escaped identifiers - #2123

Open
deepakganesh78 wants to merge 1 commit into
postcss:mainfrom
deepakganesh78:fix/issue1349-escaped-identifiers
Open

deepakganesh78 wants to merge 1 commit into
postcss:mainfrom
deepakganesh78:fix/issue1349-escaped-identifiers

Conversation

@deepakganesh78

Copy link
Copy Markdown

Fixes #1349

Reproduction

Before this change, tokenizing the issue examples split one escaped identifier into many word tokens:

const tokenizer = require('./lib/tokenize')
const { Input } = require('./lib/postcss')
let input = String.raw`C\(\#0280ae\)`

Current main produced C, \(, \#, 0280ae, \) as separate words instead of one identifier token.

Root cause

The word scanner stopped at every backslash because RE_WORD_END treats \ as a word boundary. The dedicated backslash branch consumed only the escape sequence itself, so escaped punctuation inside identifiers could never stay attached to the surrounding identifier text.

Fix

Reuse the existing escape-consumption rules while scanning word endings. When the scanner reaches a valid CSS escape, it skips over that escape and continues the same word token; invalid escapes keep the previous token boundaries. This also keeps consecutive hexadecimal escapes in the same identifier token.

Compatibility

No new dependencies. Parsing/stringification behavior is unchanged for normal CSS; the tokenizer now matches CSS identifier semantics for valid escaped punctuation.

Validation

  • node -r ts-node/register/transpile-only test/tokenize.test.js: 34/34 passed.
  • Regression test with lib/tokenize.js reverted: failed as expected (keeps escaped punctuation in identifiers).
  • pnpm run test:coverage: 682/682 unit tests passed.
  • pnpm run test:lint: passed with 0 errors and one existing warning in test/visitor.test.ts.
  • pnpm run test:types: passed.
  • pnpm run test:version: passed.
  • pnpm run test:integration: 33 real-world CSS fixtures passed.
  • pnpm run test:size: passed, 16.37 kB / 16.5 kB.

Note: on Windows, raw pnpm test fails before running because the package script uses POSIX FORCE_COLOR=1; I ran the equivalent test:* scripts individually with $env:FORCE_COLOR=1.

Keep valid CSS escapes inside the surrounding word token so escaped punctuation in identifiers is not split into separate tokens.

Fixes postcss#1349

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ai

ai commented Aug 2, 2026

Copy link
Copy Markdown
Member
  1. Such changes could be breaking changes. We can move it to PostCSS 9.0.
  2. Do you know any use case of this new AST? How parsers will use it later?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Handling escape symbols

3 participants