Repository navigation
Rewrite zipimport from scratch #69897
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2018-11-05.13:48:52.968>
created_at = <Date 2015-11-23.16:28:40.454>
labels = ['interpreter-core', 'type-feature', '3.8']
title = 'Rewrite zipimport from scratch'
updated_at = <Date 2018-11-05.13:48:52.967>
user = 'https://github.com/brettcannon'bugs.python.org fields:
activity = <Date 2018-11-05.13:48:52.967>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-05.13:48:52.968>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2015-11-23.16:28:40.454>
creator = 'brett.cannon'
dependencies = []
files = ['45800', '45814']
hgrepos = []
issue_num = 25711
keywords = ['patch']
message_count = 33.0
messages = ['255183', '255198', '255202', '255224', '256161', '256171', '256207', '257360', '257609', '282731', '282776', '282777', '282871', '282876', '282938', '283130', '299705', '304484', '304531', '306556', '306610', '307524', '316536', '321267', '321289', '323750', '324445', '324453', '324456', '325672', '325706', '325771', '329294']
nosy_count = 13.0
nosy_names = ['twouters', 'barry', 'brett.cannon', 'gregory.p.smith', 'ncoghlan', 'pmpp', 'eric.snow', 'serhiy.storchaka', 'diana', 'superluser', 'byrnes', 'Decorater', 'jarondl']
pr_nums = ['4023', '6809', '9404', '9406', '4470']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue25711'
versions = ['Python 3.8']
Activity
brettcannon commentedon Nov 23, 2015
No one wants to work on zipimport, and yet it's full of bugs. It needs a rewrite so that it's more maintainable. An idea floated at PyCon 2015 was to writing the zip-reading code in C and to keep it as simple as possible -- e.g., don't worry about supporting comments, etc. -- and then write the rest of the code in importlib, making maintenance much easier.
All of the various zipimport bugs should be made dependent on this issue as unless they are critical flaws I doubt they will get fixed without the rewrite.
serhiy-storchaka commentedon Nov 23, 2015
FYI I'm at the early stage of rewriting zipimport in Python.
brettcannon commentedon Nov 23, 2015
Are you writing it in such a way that it can be bootstspped in with importlib so the stslib can be loaded from it?
serhiy-storchaka commentedon Nov 23, 2015
It was my intention.
superluser commentedon Dec 9, 2015
Serhiy, how far along are you on this? I have a wip from this summer that I could finish over the holidays.
serhiy-storchaka commentedon Dec 10, 2015
I were on very early stage, and stopped this work few weeks ago in favor of other issues. I would be glad to make a review of your work when you have finished it Rose.
vstinner commentedon Dec 11, 2015
Can you both publish your WIP work?
brettcannon commentedon Jan 2, 2016
What are people's statuses on their various attempts? Since this is going to block my importlib.resources work I will do the work myself or work directly with someone in order to make sure this gets done.
superluser commentedon Jan 6, 2016
Sorry for the late response. I didn't have much time over the holidays. I think I better let someone else take this one.
serhiy-storchaka commentedon Dec 8, 2016
Here is preliminary translation of zipimport to Python. It is not frozen and imports other modules. I tried to keep the implementation close to C implementation. As a consequence, some raised exceptions look arbitrary.
serhiy-storchaka commentedon Dec 9, 2016
Got rid of dependencies from os, stat and encodings.
vstinner commentedon Dec 9, 2016
Technically, will it be possible to freeze it? It seems useful to keep the ability to put the whole stdlib into a single ZIP. Using a ZIP is sometimes suggested to avoid fstat() on disk for example, to speedup Python startup.
But I also understand that the C code is painful to maintain and update.
Anyway, nice job Serhiy :-)
10 remaining items
serhiy-storchaka commentedon May 14, 2018
PR 6809 freezes zipimport.py, adds changes made in zipimport.c since writing the initial Python implementation, and fixes few bugs exposed in the frozen module. Seems Python now works with the zipped stdlib.
serhiy-storchaka commentedon Jul 8, 2018
Could anybody please make a review? This is just a first step, we need to do it before making other steps: implementing more modern import API, supporting large ZIP files and ZIP files with comments, implementing loading binary extensions from ZIP files, etc.
brettcannon commentedon Jul 8, 2018
I'm planning to review the PR at some point.
On Sun, Jul 8, 2018, 00:42 Serhiy Storchaka, <report@bugs.python.org> wrote:
serhiy-storchaka commentedon Aug 19, 2018
Fixed issues on Windows.
brettcannon commentedon Aug 31, 2018
I think Serhiy's PR is basically done, so now the question is do we want to merge it in and drop the C code? ;)
I obviously say yes because this is I/O-bound code and so the switch shouldn't be enough of a performance hit to warrant blocking the gain in maintainability long-term (especially if we try to clean the module up slowly).
Barry, since I know you work with zip files a lot at work did you want to check to make sure perf won't be an issue for your use-case?
warsaw commentedon Sep 1, 2018
Not sure if I'll have time before the core sprints to check the implementation with shiv, but I can give it a try then. Do you mind waiting until then?
brettcannon commentedon Sep 1, 2018
I'm personally in no rush and I assume Serhiy isn't either with 3.8 cut-off quite a ways out, so I see no rush.
serhiy-storchaka commentedon Sep 18, 2018
New changeset 79d1c2e by Serhiy Storchaka in branch 'master':
bpo-25711: Rewrite zipimport in pure Python. (GH-6809)
79d1c2e
serhiy-storchaka commentedon Sep 19, 2018
New changeset 9da3961 by Serhiy Storchaka in branch 'master':
bpo-25711: Move _ZipImportResourceReader from importlib to zipimport. (GH-9406)
9da3961
serhiy-storchaka commentedon Sep 19, 2018
New changeset b2984ab by Serhiy Storchaka in branch 'master':
bpo-25711: Remove outdated zipimport tests. (GH-9404)
b2984ab
ncoghlan commentedon Nov 5, 2018
Noticed this was still open when reviewing Elvis's zipimport patch for bpo-34022.
Given the Python implementation has been merged, should we close this as resolved, and open new issues for any further changes (performance or otherwise)?