Skip to content

Deprecate the unused CodeObject flag CO_NESTED #96811

Closed as not planned
Closed as not planned
Deprecate the unused CodeObject flag CO_NESTED#96811
Labels
3.12only security fixes

Description

@iritkatriel
iritkatriel
opened on Sep 13, 2022

This flag is not used by the interpreter. However, it is documented (as an implementation detail) in the inspect doc at https://docs.python.org/3.12/library/inspect.html.

We should consider deprecating it.

Activity

  1. iritkatriel
    added
    type-featureA feature request or enhancement
    3.12only security fixes
    and removed
    type-featureA feature request or enhancement
    on Sep 13, 2022

  2. carljm commented on Sep 13, 2022

    @carljm
    carljm
    on Sep 13, 2022
    Member

    Looking for third-party uses, I found https://github.com/pytorch/pytorch/blob/cfb9d0d23314fd28be118b6ca280ded55364e71c/torch/utils/data/datapipes/utils/common.py#L110 and https://github.com/catboost/catboost/blob/dfc3f04050ef9acc3f070f38fe363fec93a3bc8d/contrib/python/joblib/joblib/externals/loky/cloudpickle_wrapper.py#L70

    Both appear to be detecting nested functions in order to avoid trying to pickle them, so checking for freevars wouldn't be a suitable alternative in these cases. Checking for "<locals>" in fn.__qualname__ would be.

  3. hugovk commented on Sep 14, 2022

    @hugovk
    hugovk
    on Sep 14, 2022
    Member

    Using Victor's https://github.com/vstinner/misc/blob/main/cpython/search_pypi_top.py on the top 5k PyPI packages as of 2022-06-12 (or rather the 4,768 that have sdists), it found 90 matching lines in 13 projects.

    They're mostly typeshed definitions.

    Full output 
    $ python ~/github/misc/cpython/search_pypi_top.py . CO_NESTED --quiet
./pydevd-pycharm-222.2889.11.tar.gz: pydevd-pycharm-222.2889.11/_pydev_imps/_pydev_inspect.py: CO_NESTED, CO_GENERATOR, CO_NOFREE = 0x10, 0x20, 0x40
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv/lib/python3.8/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv/lib/python3.8/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv/lib/python3.8/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv/lib/python3.8/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv/lib64/python3.8/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv/lib64/python3.8/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv/lib64/python3.8/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv/lib64/python3.8/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv-py3.8/lib/python3.8/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv-py3.8/lib/python3.8/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv-py3.8/lib/python3.8/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv-py3.8/lib/python3.8/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv-py3.8/lib64/python3.8/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv-py3.8/lib64/python3.8/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv-py3.8/lib64/python3.8/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./hexbytes-0.2.2.tar.gz: hexbytes-0.2.2/venv-py3.8/lib64/python3.8/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./jedi-0.18.1.tar.gz: jedi-0.18.1/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./jedi-0.18.1.tar.gz: jedi-0.18.1/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-utils-2.0.0.tar.gz: eth-utils-2.0.0/.tox/lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-utils-2.0.0.tar.gz: eth-utils-2.0.0/.tox/lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-utils-2.0.0.tar.gz: eth-utils-2.0.0/venv-eu/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-utils-2.0.0.tar.gz: eth-utils-2.0.0/venv-eu/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-utils-2.0.0.tar.gz: eth-utils-2.0.0/venv-eu/lib/python3.9/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-utils-2.0.0.tar.gz: eth-utils-2.0.0/venv-eu/lib/python3.9/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-utils-2.0.0.tar.gz: eth-utils-2.0.0/venv-eu-200/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-utils-2.0.0.tar.gz: eth-utils-2.0.0/venv-eu-200/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-utils-2.0.0.tar.gz: eth-utils-2.0.0/venv-eu-200/lib/python3.9/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-utils-2.0.0.tar.gz: eth-utils-2.0.0/venv-eu-200/lib/python3.9/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/.tox/lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/@python2/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/.tox/lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/inspect.pyi: CO_NESTED: int
./coverage-6.4.1.tar.gz: coverage-6.4.1/lab/show_pyc.py: ('CO_NESTED',                   0x00010),
./mypy-0.961.tar.gz: mypy-0.961/mypy/typeshed/stdlib/@python2/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/venv-abi-ii/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./mypy-0.961.tar.gz: mypy-0.961/mypy/typeshed/stdlib/inspect.pyi: "CO_NESTED",
./mypy-0.961.tar.gz: mypy-0.961/mypy/typeshed/stdlib/inspect.pyi: CO_NESTED: Literal[16]
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/venv-abi-ii/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/venv-abi-ii/lib/python3.9/site-packages/mypy/typeshed/stdlib/@python2/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/venv-abi-ii/lib/python3.9/site-packages/mypy/typeshed/stdlib/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/venv-eabi/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/venv-eabi/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/venv-eabi/lib/python3.9/site-packages/mypy/typeshed/stdlib/@python2/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/venv-eabi/lib/python3.9/site-packages/mypy/typeshed/stdlib/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/venv-eabi-3/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/venv-eabi-3/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/venv-eabi-3/lib/python3.9/site-packages/mypy/typeshed/stdlib/@python2/inspect.pyi: CO_NESTED: int
./eth_abi-3.0.0.tar.gz: eth_abi-3.0.0/venv-eabi-3/lib/python3.9/site-packages/mypy/typeshed/stdlib/inspect.pyi: CO_NESTED: int
./pytype-2022.6.6.tar.gz: pytype-2022.6.6/pytype/pyc/loadmarshal.py: CO_NESTED = 0x0010
./eth-rlp-0.3.0.tar.gz: eth-rlp-0.3.0/.tox/lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./pytype-2022.6.6.tar.gz: pytype-2022.6.6/pytype/typeshed/stdlib/@python2/inspect.pyi: CO_NESTED: int
./eth-rlp-0.3.0.tar.gz: eth-rlp-0.3.0/.tox/lint/lib/python3.9/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./pytype-2022.6.6.tar.gz: pytype-2022.6.6/pytype/typeshed/stdlib/inspect.pyi: "CO_NESTED",
./pytype-2022.6.6.tar.gz: pytype-2022.6.6/pytype/typeshed/stdlib/inspect.pyi: CO_NESTED: Literal[16]
./eth-rlp-0.3.0.tar.gz: eth-rlp-0.3.0/venv-erlp/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-rlp-0.3.0.tar.gz: eth-rlp-0.3.0/venv-erlp/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/.tox/lint/lib/python3.8/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/.tox/lint/lib/python3.8/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-rlp-0.3.0.tar.gz: eth-rlp-0.3.0/venv-erlp/lib/python3.9/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-rlp-0.3.0.tar.gz: eth-rlp-0.3.0/venv-erlp/lib/python3.9/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv/lib/python3.8/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv/lib/python3.8/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv/lib/python3.8/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv/lib/python3.8/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv/lib64/python3.8/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv/lib64/python3.8/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv/lib64/python3.8/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv/lib64/python3.8/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv-py3.8/lib/python3.8/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv-py3.8/lib/python3.8/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv-py3.8/lib/python3.8/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv-py3.8/lib/python3.8/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv-py3.8/lib64/python3.8/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv-py3.8/lib64/python3.8/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv-py3.8/lib64/python3.8/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-hash-0.3.2.tar.gz: eth-hash-0.3.2/venv-py3.8/lib64/python3.8/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./joblib-1.1.0.tar.gz: joblib-1.1.0/joblib/externals/loky/cloudpickle_wrapper.py: need_wrap |= getattr(func_code, "co_flags", 0) & inspect.CO_NESTED
./xdis-6.0.4.tar.gz: xdis-6.0.4/xdis/__init__.py: CO_NESTED,
./xdis-6.0.4.tar.gz: xdis-6.0.4/xdis/__init__.py: "CO_NESTED",
./eth-account-0.6.1.tar.gz: eth-account-0.6.1/venv-056/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-account-0.6.1.tar.gz: eth-account-0.6.1/venv-056/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-account-0.6.1.tar.gz: eth-account-0.6.1/venv-056/lib/python3.9/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-account-0.6.1.tar.gz: eth-account-0.6.1/venv-056/lib/python3.9/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-account-0.6.1.tar.gz: eth-account-0.6.1/venv-ea-060/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-account-0.6.1.tar.gz: eth-account-0.6.1/venv-ea-060/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-account-0.6.1.tar.gz: eth-account-0.6.1/venv-ea-060/lib/python3.9/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-account-0.6.1.tar.gz: eth-account-0.6.1/venv-ea-060/lib/python3.9/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-account-0.6.1.tar.gz: eth-account-0.6.1/venv-ea-eabi/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-account-0.6.1.tar.gz: eth-account-0.6.1/venv-ea-eabi/lib/python3.9/site-packages/jedi/third_party/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int
./eth-account-0.6.1.tar.gz: eth-account-0.6.1/venv-ea-eabi/lib/python3.9/site-packages/mypy/typeshed/stdlib/2/inspect.pyi: CO_NESTED: int
./eth-account-0.6.1.tar.gz: eth-account-0.6.1/venv-ea-eabi/lib/python3.9/site-packages/mypy/typeshed/stdlib/3/inspect.pyi: CO_NESTED: int

Time: 0:00:16.292883
Found 90 matching lines in 13 projects

    The non-typeshed results:

    $ python ~/github/misc/cpython/search_pypi_top.py . CO_NESTED --quiet | grep -v typeshed
./pydevd-pycharm-222.2889.11.tar.gz: pydevd-pycharm-222.2889.11/_pydev_imps/_pydev_inspect.py: CO_NESTED, CO_GENERATOR, CO_NOFREE = 0x10, 0x20, 0x40
./coverage-6.4.1.tar.gz: coverage-6.4.1/lab/show_pyc.py: ('CO_NESTED',                   0x00010),
./pytype-2022.6.6.tar.gz: pytype-2022.6.6/pytype/pyc/loadmarshal.py: CO_NESTED = 0x0010
./joblib-1.1.0.tar.gz: joblib-1.1.0/joblib/externals/loky/cloudpickle_wrapper.py: need_wrap |= getattr(func_code, "co_flags", 0) & inspect.CO_NESTED
./xdis-6.0.4.tar.gz: xdis-6.0.4/xdis/__init__.py: CO_NESTED,
./xdis-6.0.4.tar.gz: xdis-6.0.4/xdis/__init__.py: "CO_NESTED",

Time: 0:00:15.730672
Found 90 matching lines in 13 projects

  4. vstinner commented on Sep 14, 2022

    @vstinner
    vstinner
    on Sep 14, 2022
    Member

    The non-typeshed results

    None of these projects set the flag, only cloudpickle_wrapper.py of joblib checks if the flag is set or not.

  5. vstinner commented on Sep 14, 2022

    @vstinner
    vstinner
    on Sep 14, 2022
    Member

    This flag is not used by the interpreter.

    IMO in this case, it's fine to deprecate the flag. Thanks for checking top 5000 PyPI projects, that's useful to take a decision.

  6. iritkatriel commented on Sep 14, 2022

    @iritkatriel
    iritkatriel
    on Sep 14, 2022
    MemberAuthor

    Both appear to be detecting nested functions in order to avoid trying to pickle them, so checking for freevars wouldn't be a suitable alternative in these cases. Checking for "<locals>" in fn.__qualname__ would be.

    If people need this, and that's the best alternative then maybe we should keep the flag?

  7. vstinner commented on Sep 14, 2022

    @vstinner
    vstinner
    on Sep 14, 2022
    Member

    Oh, right. In Python 3.12, CO_NESTED is still sets properly: it can be used to check if a function was declared inside another function.

    import inspect
def f():
    def g(): pass
    return g

def is_nested(func):
    code = func.__code__
    print(f"{func.__name__} is nested?", bool(code.co_flags & inspect.CO_NESTED))

g = f()
is_nested(f)
is_nested(g)

    Python 3.12 output:

    f is nested? False
g is nested? True

    Is there an alternative without testing the CO_NESTED flag?

  8. carljm commented on Sep 14, 2022

    @carljm
    carljm
    on Sep 14, 2022
    Member

    Is there an alternative without testing the CO_NESTED flag?

    I mentioned both alternatives that I know of above. The only real equivalent AFAIK is to check "<locals>" in func.__qualname__, which should give the same result in all cases I'm aware of. A semi-equivalent is to check if the function has freevars, if what you really care about is whether it closes over any values, but some nested functions don't have freevars.

  9. iritkatriel commented on Sep 16, 2022

    @iritkatriel
    iritkatriel
    on Sep 16, 2022
    MemberAuthor

    I think we should not remove it. It's used, there is no good alternative, and it's not in anybody's way.

  10. iritkatriel
    closed this as not plannedon Sep 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.12only security fixes

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions