Repository navigation
Deprecate the unused tokenize._compile function #143009
Description
Feature or enhancement
Proposal:
Remove the unused EDIT: Deprecate the unused tokenize._compile function.tokenize._compile function.
It very much looks like the internal tokenize._compile function hasn't been needed since GH-104323 when it was initially removed, but it was mechanically brought back to tokenize in GH-104722 to fix GH-104719.
From a quick search, it seems that, prior to GH-104323, tokenize._compile was only used internally in tokenize._tokenize.
Most of the fairly fresh code I skimmed through on GitHub operating on tokenize._compile properly checks if Python is version <3.10 / tokenize._compile exists.
Except CheetahTemplate3, so removing tokenize._compile immediately could break them -- we need to deprecate first.
In terms of benefits of removing this at some point: it wouldn't necessarily improve the import time of tokenize as long as re still imports functools to cache template compilation. Therefore, removing tokenize._compile in the future is just a small cleanup that yields no other benefits than smaller, less bloated code.
Has this already been discussed elsewhere?
This is a minor feature, which does not need previous discussion elsewhere
Links to previous discussion of this feature:
No response
Activity
johnslavik commentedon Dec 20, 2025
@ZeroIntensity
stdlibtopic-parserCC-ing @pablogsal @Yhg1s @lysnikolaou
johnslavik commentedon Dec 20, 2025
Talked with @ZeroIntensity, the general guideline is to avoid breaking things, so I'll rephrase this issue to instead propose deprecating first.
[-]Remove the unused `tokenize._compile` function[/-][+]Deprecate the unused `tokenize._compile` function[/+]pablogsal commentedon Dec 20, 2025
I am not against it but I think is weird to deprecate a private function and the gain for maintainance is very little as we don't have any good reason for doing it right now (security, bugs...etc)
johnslavik commentedon Dec 20, 2025
I see where you're coming from. Do you think it's better to wait for a good reason to deprecate or remove, instead of doing it right now, when there is no immediate benefit?
pablogsal commentedon Dec 20, 2025
Pretty much. In general these kind of changes have more chances to annoy some users then to really improve the maintainance of the code. It's a bit unfortunate that people rely on private functions but at this stage there is no much obvious benefit and I just see risk
johnslavik commentedon Dec 20, 2025
I can see the problem. I'll close this -- I also think it's not worth the risk. Thanks @pablogsal!