Skip to content

gh-119118: Fix performance regression in tokenize module #119615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pablogsal merged 7 commits into from
May 28, 2024
Merged

gh-119118: Fix performance regression in tokenize module #119615

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
66f9385
gh-119118: Fix performance regression in tokenize module
lysnikolaou May 20, 2024
143a420
📜🤖 Added by blurb_it.
blurb-it[bot] May 28, 2024
8318ab8
Update Python/Python-tokenize.c
lysnikolaou May 28, 2024
566adc3
WIP: strcmp lines (need to fix) and use line buffer for end col offset
lysnikolaou May 28, 2024
858eff1
Get rid of strcmp
lysnikolaou May 28, 2024
ab34370
Make NEWS linter happy
lysnikolaou May 28, 2024
b97c51e
Merge branch 'main' into performance-tokenize
pablogsal May 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Make NEWS linter happy
  • Loading branch information
@lysnikolaou
lysnikolaou committed May 28, 2024
commit ab3437096a729c05cfbffd4ddd7a86baf697efb3
3 changes: 2 additions & 1 deletion Misc/NEWS.d/next/Library/2024-05-28-12-15-03.gh-issue-119118.FMKz1F.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
Fix performance regression in the :mod:`tokenize` module by caching the `line` token attribute and calculating the column offset more efficiently.
Fix performance regression in the :mod:`tokenize` module by caching the ``line``
token attribute and calculating the column offset more efficiently.