Skip to content

Can't use lib2to3 with embeddable zip file. #69148

Closed
Closed
Can't use lib2to3 with embeddable zip file.#69148
Labels
3.7 (EOL)end of lifeperformancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directory

Description

@OlegN
OlegN
mannequin
opened on Aug 29, 2015
BPO 24960
Nosy @gpshead, @pfmoore, @ncoghlan, @benjaminp, @dstufft, @ppinard, @Lukas00500
PRs
  • bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files #4977
  • [3.6] bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (GH-4977) #4979
  • [2.7] bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (GH-4977) #4980
  • make PatternCompiler use the packaged grammar if possible (more bpo-24960) #5034
  • [3.6] make PatternCompiler use the packaged grammar if possible (more bpo-24960) (GH-5034) #5036
  • [2.7] make PatternCompiler use the packaged grammar if possible (more bpo-24960) (GH-5034) #5037
    		•
    Files
  • pgen.py
  • pgen.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-12-22.20:52:37.021>
created_at = <Date 2015-08-29.17:36:22.794>
labels = ['3.7', 'library', 'performance']
title = "Can't use lib2to3 with embeddable zip file."
updated_at = <Date 2018-04-03.21:10:50.278>
user = 'https://bugs.python.org/OlegN'

    bugs.python.org fields:

    activity = <Date 2018-04-03.21:10:50.278>
actor = 'Lukasz00500'
assignee = 'none'
closed = True
closed_date = <Date 2017-12-22.20:52:37.021>
closer = 'benjamin.peterson'
components = ['Library (Lib)']
creation = <Date 2015-08-29.17:36:22.794>
creator = 'Oleg N'
dependencies = []
files = ['46866', '46869']
hgrepos = []
issue_num = 24960
keywords = ['patch']
message_count = 18.0
messages = ['249329', '249804', '255441', '269765', '269779', '285753', '285756', '292529', '293780', '293855', '294105', '308940', '308941', '308942', '309153', '309154', '309155', '314896']
nosy_count = 13.0
nosy_names = ['gregory.p.smith', 'paul.moore', 'ncoghlan', 'benjamin.peterson', 'dstufft', 'Marcus.Smith', 'Oleg N', 'Florian Hassanen', 'fourplusone', 'S\xc3\xa9bastien Taylor', 'ppinard', 'kirk.d.sayre', 'Lukasz00500']
pr_nums = ['4977', '4979', '4980', '5034', '5036', '5037']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue24960'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7']

    Activity

    1. OlegN commented on Aug 29, 2015

      @OlegN
      OlegNmannequin
      on Aug 29, 2015
      MannequinAuthor

      Error: [Errno 2] No such file or directory: '...\\python35.zip\\lib2to3\\Grammar.txt'. Python35.zip from this archive: https://www.python.org/ftp/python/3.5.0/python-3.5.0rc2-embed-amd64.zip. Path is correct.

    2. OlegN
      added
      stdlibStandard Library Python modules in the Lib/ directory
      performancePerformance or resource usage
      on Aug 29, 2015

    3. dstufft commented on Sep 4, 2015

      @dstufft
      dstufft
      on Sep 4, 2015
      Member

      This looks more like lib2to3 doesn't support running from a .zip archive.

    4. FlorianHassanen commented on Nov 26, 2015

      @FlorianHassanen
      FlorianHassanenmannequin
      on Nov 26, 2015
      Mannequin

      got this traceback:

      Traceback (most recent call last):
  File "setup.py", line 169, in <module>
    dist = setuptools.setup(**setup_params)
  File "distutils\core.py", line 134, in setup
  File "...\setuptools\setuptools\dist.py", line 299, in parse_command_line
    result = _Distribution.parse_command_line(self)
  File "distutils\dist.py", line 469, in parse_command_line
  File "...\setuptools\setuptools\dist.py", line 679, in handle_display_options
    return _Distribution.handle_display_options(self, option_order)
  File "distutils\dist.py", line 672, in handle_display_options
  File "...\setuptools\setuptools\dist.py", line 439, in print_commands
    cmdclass = ep.resolve()
  File "...\setuptools\pkg_resources\__init__.py", line 2383, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "...\setuptools\setuptools\command\build_py.py", line 10, in <module>
    from setuptools.lib2to3_ex import Mixin2to3
  File "...\setuptools\setuptools\lib2to3_ex.py", line 12, in <module>
    from lib2to3.refactor import RefactoringTool, get_fixers_from_package
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "lib2to3\refactor.py", line 27, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "lib2to3\fixer_util.py", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "lib2to3\pygram.py", line 32, in <module>
  File "lib2to3\pgen2\driver.py", line 121, in load_grammar
  File "lib2to3\pgen2\pgen.py", line 385, in generate_grammar
  File "lib2to3\pgen2\pgen.py", line 15, in __init__
FileNotFoundError: [Errno 2] No such file or directory: '...\\python-3.5.0-embed-amd64\\python35.zip\\lib2to3\\Grammar.txt'

      triggered e.g. by "python setup.py --help-commands" (when building setuptools)

      workaround is to extract "...\python-3.5.0-embed-amd64\python35.zip" into a __directory__ named exactly the same (hence a directory named "python35.zip" which replaces the original zip file)

    5. fourplusone commented on Jul 3, 2016

      @fourplusone
      fourplusonemannequin
      on Jul 3, 2016
      Mannequin

      The problem is indeed that lib2to3 fails to load the Grammar.txt file if it is located inside a .zip archive.

      I think one way to fix it is using of the "get_data" function of the loader https://docs.python.org/3/library/importlib.html#importlib.abc.ResourceLoader.get_data

      However, this would make caching the Grammar more difficult, as one cannot write back the pickled dict to a zip file

      Alternatively one could generate an importable *.pyc file from the grammar during the build process

    6. ncoghlan commented on Jul 4, 2016

      @ncoghlan
      ncoghlan
      on Jul 4, 2016
      Contributor

      (issue title updated to indicate this appears to be lib2to3 related and added Benjamin Peterson to the nosy list)

      The build process already generates a _sysconfigdata.py file in order to pass along build time details to the sysconfig module in a way that isn't dependent on having otherwise build-time only files accessible at runtime.

      Doing something similar with a _lib2to3grammar.py file seems like it could be a reasonable approach to making lib2to3 zip archive friendly, but I'm not familiar enough with pgen2's internals to say how feasible that would be.

    7. ncoghlan
      changed the title [-]Can't use pip or easy_install with embeddable zip file.[/-] [+]Can't use lib2to3 with embeddable zip file.[/+] on Jul 4, 2016

    8. pfmoore commented on Jan 18, 2017

      @pfmoore
      pfmoore
      on Jan 18, 2017
      Member

      I'm not sure I see why you'd want to run lib2to3 with the embedded distribution anyway. Surely you'd do any 2to3 conversion in a normal development installation, and then simply bundle the resulting converted scripts with your application and the embedded interpreter?

    9. SbastienTaylor commented on Jan 18, 2017

      @SbastienTaylor
      SbastienTaylormannequin
      on Jan 18, 2017
      Mannequin

      Are there existing tools for bundling the python35.zip (and adding additional packages)? I personally ran into this issue by using pip to add additional modules to my embeddable python but if there's a better way to migrate from devel to embeddable python I'd rather not have to use pip for this.

    10. ppinard commented on Apr 28, 2017

      @ppinard
      ppinardmannequin
      on Apr 28, 2017
      Mannequin

      As Sébastien Taylor, I ran into the same problem. The workaround I found was to unzip the content of python35.zip and put it in the Lib/ folder.

    11. kirkdsayre commented on May 16, 2017

      @kirkdsayre
      kirkdsayremannequin
      on May 16, 2017
      Mannequin

      I've been fighting with this issue today trying to import networkx in the embedded Python shipped with IDA-Pro. I have implemented a local fix to pgen.py that allows networkx to import and be used in IDAPython scripts. I added a check to see if a .zip archive appears in the path of the file loaded in the constructor and if so I use the zipfile package to open the archived file rather than the standard file open() function.

      I've uploaded the modified pgen.py in case any one is interested.

    12. kirkdsayre commented on May 17, 2017

      @kirkdsayre
      kirkdsayremannequin
      on May 17, 2017
      Mannequin

      Here is a more robust version of my local fix for this problem.

    13. gpshead commented on May 21, 2017

      @gpshead
      gpshead
      on May 21, 2017
      Member

      re: Paul's "I'm not sure I see why you'd want to run lib2to3 with the embedded distribution anyway. Surely you'd do any 2to3 conversion in a normal development installation"

      lib2to3 is a library useful for doing things with Python code. Not just a stand alone tool.

      Kirk: Could you please post your change as a pull request on https://github.com/python/cpython/ rather than attaching individual files here. a full file isn't a reviewable change.

      Also, Grammar.txt is loaded and saved as a cached pickle file for speed. When packaging up an embedded stdlib bundle I recommend shipping that pickle. (loaded in pgen2/grammar.py).

      both probably need to use https://docs.python.org/3.6/library/pkgutil.html#pkgutil.get_data to read the file out of the package rather than assuming it lives on a filesystem.

      using the zipfile module within lib2to3 to try and figure out when it should look somewhere other than a filesystem is the wrong approach.

    14. gpshead
      added
      3.7 (EOL)end of life
      on May 21, 2017

    15. benjaminp commented on Dec 22, 2017

      @benjaminp
      benjaminp
      on Dec 22, 2017
      Contributor

      New changeset 8a58771 by Benjamin Peterson in branch 'master':
      bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (bpo-4977)
      8a58771

    16. benjaminp commented on Dec 22, 2017

      @benjaminp
      benjaminp
      on Dec 22, 2017
      Contributor

      New changeset c1b8eb8 by Benjamin Peterson (Miss Islington (bot)) in branch '3.6':
      bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (GH-4977) (bpo-4979)
      c1b8eb8

    17. benjaminp commented on Dec 22, 2017

      @benjaminp
      benjaminp
      on Dec 22, 2017
      Contributor

      New changeset 770a802 by Benjamin Peterson in branch '2.7':
      [2.7] bpo-24960: use pkgutil.get_data in lib2to3 to read pickled grammar files (GH-4977) (bpo-4980)
      770a802

    18. benjaminp
      closed this as completedon Dec 22, 2017

    19. benjaminp commented on Dec 29, 2017

      @benjaminp
      benjaminp
      on Dec 29, 2017
      Contributor

      New changeset e5f7dcc by Benjamin Peterson in branch 'master':
      make PatternCompiler use the packaged grammar if possible (more bpo-24960) (bpo-5034)
      e5f7dcc

    20. benjaminp commented on Dec 29, 2017

      @benjaminp
      benjaminp
      on Dec 29, 2017
      Contributor

      New changeset 417f76a by Benjamin Peterson (Miss Islington (bot)) in branch '2.7':
      make PatternCompiler use the packaged grammar if possible (more bpo-24960) (GH-5034) (bpo-5037)
      417f76a

    21. benjaminp commented on Dec 29, 2017

      @benjaminp
      benjaminp
      on Dec 29, 2017
      Contributor

      New changeset 85f71aa by Benjamin Peterson (Miss Islington (bot)) in branch '3.6':
      make PatternCompiler use the packaged grammar if possible (more bpo-24960) (GH-5034) (bpo-5036)
      85f71aa

    22. Lukas00500 commented on Apr 3, 2018

      @Lukas00500
      Lukas00500mannequin
      on Apr 3, 2018
      Mannequin

      I have simmilar problem but im using python 2.7 on android. So I think I cant use this solution
      My problem:
      https://bugs.python.org/issue33208

    23. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of lifeperformancePerformance or resource usagestdlibStandard Library Python modules in the Lib/ directory

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions