New files() api from importlib_resources.

@warsaw

BPO	39791
Nosy	@warsaw, @nascheme, @nascheme, @jaraco, @ned-deily, @ambv, @aixtools, @miss-islington, @hauntsaninja, @benthayer
PRs	bpo-39791: Add files() to importlib.resources #19722 bpo-39791 native hooks for importlib.resources.files #20576 bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. #20656 bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. #20659 [3.9] bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. (GH-20659) #20661 [3.8] bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. (GH-20659) #20662 bpo-39791: Support file systems that cannot support non-ascii filenames. #20681 [3.9] bpo-39791: Support file systems that cannot support non-ascii filenames (skipping tests in that case). (GH-20681) #20695 [3.9] bpo-39791 native hooks for importlib.resources.files (GH-20576) #20703 [3.9] bpo-40924: Revert "bpo-39791 native hooks for importlib.resources.files (GH-20576)" #20760 bpo-41006: importlib.util no longer imports typing #20938 bpo-39791: importlib.abc: avoid use of deprecated abstractproperty #23715 bpo-43288: Fix test_importlib test to correctly skip #24612 bpo-43288: Fix test_importlib test to correctly skip #24612

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/jaraco'
closed_at = <Date 2020-12-09.23:47:15.564>
created_at = <Date 2020-02-29.04:11:30.709>
labels = ['type-bug', 'library', '3.9']
title = 'New `files()` api from importlib_resources.'
updated_at = <Date 2021-02-22.00:21:55.488>
user = 'https://github.com/jaraco'

bugs.python.org fields:

activity = <Date 2021-02-22.00:21:55.488>
actor = 'vstinner'
assignee = 'jaraco'
closed = True
closed_date = <Date 2020-12-09.23:47:15.564>
closer = 'jaraco'
components = ['Library (Lib)']
creation = <Date 2020-02-29.04:11:30.709>
creator = 'jaraco'
dependencies = []
files = []
hgrepos = []
issue_num = 39791
keywords = ['patch']
message_count = 17.0
messages = ['362962', '363556', '368484', '368490', '370782', '370790', '370825', '370834', '370840', '370903', '370904', '370945', '370947', '371074', '371131', '371744', '382812']
nosy_count = 9.0
nosy_names = ['barry', 'nascheme', 'nascheme', 'jaraco', 'ned.deily', 'lukasz.langa', 'Michael.Felt', 'miss-islington', 'hauntsaninja', 'benthayer']
pr_nums = ['19722', '20576', '20656', '20659', '20661', '20662', '20681', '20695', '20703', '20760', '20938', '23715', '24612', '24612']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39791'
versions = ['Python 3.9']

In the importlib_resources backport... in particular in issue 58 and merge request 76, the backport now has a new feature, a "files()" function.

Let's incorporate that functionality into importlib.resources.

The latest release, 1.3.0, includes extensibility support and has been merged with the cpython branch of the importlib_resources project. I believe that code is now synced with this project and ready to be applied here. I'm hoping benthayer can apply the changes and submit the pr.

New changeset 7f7e706 by Jason R. Coombs in branch 'master':
bpo-39791: Add files() to importlib.resources (GH-19722)
7f7e706

I've merged PR 19722. Some follow up actions I'd like to do:

Add hooks for .files() on built-in loaders.
Replace loader.get_resource_reader() with adapters around .files() for built-in loaders.

New changeset 161541a by Jason R. Coombs in branch 'master':
bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. (GH-20659)
161541a

New changeset a4fa9a9 by Miss Islington (bot) in branch '3.9':
bpo-39791: Refresh importlib.metadata from importlib_metadata 1.6.1. (GH-20659) (GH-20661)
a4fa9a9

The 'fancy' file name breaks on latin-1 character set.

aixtools@gcc119:[/home/aixtools/python/py39-3.9] a4fa9a95153a3800dea60b3029b2dcaf8a4f6acb Lib/test/test_importlib/test_main.py                   <
diff --git a/Lib/test/test_importlib/test_main.py b/Lib/test/test_importlib/test_main.py
index 42a7999..7b18c3d 100644
--- a/Lib/test/test_importlib/test_main.py
+++ b/Lib/test/test_importlib/test_main.py
@@ -246,3 +246,19 @@ class TestEntryPoints(unittest.TestCase):
         """
         with self.assertRaises(Exception):
             json.dumps(self.ep)
+
+    def test_module(self):
+        assert self.ep.module == 'value'
+
+    def test_attr(self):
+        assert self.ep.attr is None
+
+
+class FileSystem(fixtures.OnSysPath, fixtures.SiteDir, unittest.TestCase):
+    def test_unicode_dir_on_sys_path(self):
+        """
+        Ensure a Unicode subdirectory of a directory on sys.path
+        does not crash.
+        """
+        fixtures.build_files({'☃': {}}, prefix=self.site_dir)
+        list(distributions())

AIX bots fail with:

Captured traceback
==================

Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.9.edelsohn-aix-ppc64/build/Lib/test/test_importlib/test_main.py", line 263, in test_unicode_dir_on_sys_path
    fixtures.build_files({'\u2603': {}}, prefix=self.site_dir)
  File "/home/shager/cpython-buildarea/3.9.edelsohn-aix-ppc64/build/Lib/test/test_importlib/fixtures.py", line 202, in build_files
    full_name.mkdir()
  File "/home/shager/cpython-buildarea/3.9.edelsohn-aix-ppc64/build/Lib/pathlib.py", line 1309, in mkdir
    self._accessor.mkdir(self, mode)
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2603' in position 17: ordinal not in range(256)

Test report
===========

Failed tests:

test_importlib

Thanks for the report Michael. I'm trying to figure out the best way to address the issue. That test is shared with importlib_metadata, so needs to run without CPython's test suite fixtures, such as the ones that generate non-ascii filenames. I'm tempted to just attempt to create the fixture and skip if the fixture fails. An alternate approach might be to attempt to load CPython's fixture, skip if that value is None, and fallback to the snowman otherwise.

See #64880 for a proposed fix. I've scheduled the build bots to run the patch. Will build bots prove the fix? If not, can you test the patch in the same environment where it was discovered?

New changeset 2efe18b by Jason R. Coombs in branch 'master':
bpo-39791: Support file systems that cannot support non-ascii filenames (skipping tests in that case). (bpo-20681)
2efe18b

New changeset 71f5016 by Miss Islington (bot) in branch '3.9':
bpo-39791: Support file systems that cannot support non-ascii filenames (skipping tests in that case). (GH-20681)
71f5016

New changeset 843c277 by Jason R. Coombs in branch 'master':
bpo-39791 native hooks for importlib.resources.files (GH-20576)
843c277

New changeset 9cf1be4 by Miss Islington (bot) in branch '3.9':
bpo-39791 native hooks for importlib.resources.files (GH-20576)
9cf1be4

Note that the most recent commits have introduced a critical regression in importlib.resources.path() that breaks the certifi package from PyPI and presumably other users of path(). See bpo-40924.

New changeset ce5e6f0 by Łukasz Langa in branch '3.9':
[3.9] bpo-40924: Revert "bpo-39791 native hooks for importlib.resources.files (GH-20576)" (bpo-20760)
ce5e6f0

bpo-39791 native hooks for importlib.resources.files (GH-20576)
843c277

This change introduced bpo-41007 "test_importlib logs ResourceWarning: test_path.CommonTests.test_importing_module_as_side_effect()" regression.

This issue was implemented, just not as fully as I'd have hoped. Still lacking is native support for .files on the built-in package providers and removing the legacy APIs or at least configuring them to rely on the files API, but as far as supplying the files API, this work is done.

Navigation Menu

Repository navigation

New `files()` api from importlib_resources. #83972

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

New files() api from importlib_resources. #83972

Description

Activity

jaraco commented on Feb 29, 2020

jaraco commented on Mar 6, 2020

jaraco commented on May 8, 2020

jaraco commented on May 8, 2020

jaraco commented on Jun 5, 2020

jaraco commented on Jun 5, 2020

aixtools commented on Jun 6, 2020

jaraco commented on Jun 6, 2020

jaraco commented on Jun 6, 2020

jaraco commented on Jun 7, 2020

miss-islington commented on Jun 7, 2020

jaraco commented on Jun 8, 2020

miss-islington commented on Jun 8, 2020

ned-deily commented on Jun 9, 2020

ambv commented on Jun 9, 2020

vstinner commented on Jun 17, 2020

jaraco commented on Dec 9, 2020

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

New `files()` api from importlib_resources. #83972