Don't use pkg_resources.get_distribution(..).version

Environment details

OS type and version: Alpine Linux edge
Python version: 3.8
pip version: Irrelevant, pure system packages used
google-api-core version: 1.16.0

Issue

I'm trying to run the tests with the end goal being packaging this for Alpine Linux.
All the dependencies are installed and I'm running the tests with PYTHONPATH="$PWD/build/lib" pytest so I can run the tests without having the package installed yet.

I'm probably just doing something wrong, but every test file fails with the following:

_____________________________________________ ERROR collecting tests/unit/test_bidi.py ______________________________________________
tests/unit/test_bidi.py:24: in <module>
    from google.api_core import bidi
build/lib/google/api_core/__init__.py:23: in <module>
    __version__ = get_distribution("google-api-core").version
/usr/lib/python3.8/site-packages/pkg_resources/__init__.py:482: in get_distribution
    dist = get_provider(dist)
/usr/lib/python3.8/site-packages/pkg_resources/__init__.py:358: in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
/usr/lib/python3.8/site-packages/pkg_resources/__init__.py:901: in require
    needed = self.resolve(parse_requirements(requirements))
/usr/lib/python3.8/site-packages/pkg_resources/__init__.py:787: in resolve
    raise DistributionNotFound(req, requirers)
E   pkg_resources.DistributionNotFound: The 'google-api-core' distribution was not found and is required by the application

I think it may have been picking up the dev copy (not the one in build). I can repro if I move to another directory after build :)

❯ python -m venv venv; source venv/bin/activate
❯ pip freeze
❯ python setup.py build
❯ export PYTHONPATH="${PWD}/build/lib"
❯ cd build


❯ python
Python 3.8.5 (default, Sep 21 2020, 13:30:58)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("/Users/crwilcox/workspace/python-api-core/build/lib")

>>> import pkg_resources
>>> pkg_resources.get_distribution("google-api-core")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/crwilcox/workspace/python-api-core/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 482, in get_distribution
    dist = get_provider(dist)
  File "/Users/crwilcox/workspace/python-api-core/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 358, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/crwilcox/workspace/python-api-core/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/crwilcox/workspace/python-api-core/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'google-api-core' distribution was not found and is required by the application

However, worth noting, I also can't import the package at all, I can get the error.

❯ python
Python 3.8.5 (default, Sep 21 2020, 13:30:58)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("/Users/crwilcox/workspace/python-api-core/build/lib")
>>> from google import api_core
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'api_core' from 'google' (unknown location)

Is the package built (copied) into ~/workspace/python-api-core/build/lib?

Yes.

❯ pwd
/Users/crwilcox/workspace/python-api-core/build/lib/google
❯ ls
__init__.py __pycache__ api_core

I am looking into moving us to not using pkg_resources. Though it is part of namespacing (we do have a fallback to pkgutil)

Also, in order to change this we need to modify our release tooling as it assumes version is specified in setup.py.

Not meaning to cross thread the issue, but we do seem to have a repro in AppEngine Standard for Python 2.7

Vendoring google-cloud-datastore into the basic quickstart (hello world) the runtime will fail similarly.

Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 238, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 309, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 83, in LoadObject
    obj = __import__(path[0])
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/main.py", line 17, in <module>
    from google.cloud import datastore
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/lib/google/cloud/datastore/__init__.py", line 62, in <module>
    from google.cloud.datastore.batch import Batch
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/lib/google/cloud/datastore/batch.py", line 24, in <module>
    from google.cloud.datastore import helpers
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/lib/google/cloud/datastore/helpers.py", line 29, in <module>
    from google.cloud.datastore_v1.proto import datastore_pb2
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/lib/google/cloud/datastore_v1/__init__.py", line 17, in <module>
    from google.cloud.datastore_v1 import types
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/lib/google/cloud/datastore_v1/types.py", line 25, in <module>
    from google.api_core.protobuf_helpers import get_messages
  File "/base/data/home/apps/s~crwilcox-test/20201001t151852.429923747926530875/lib/google/api_core/__init__.py", line 23, in <module>
    __version__ = get_distribution("google-api-core").version
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/third_party/setuptools-36.6.0/pkg_resources/__init__.py", line 562, in get_distribution
    dist = get_provider(dist)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/third_party/setuptools-36.6.0/pkg_resources/__init__.py", line 436, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/third_party/setuptools-36.6.0/pkg_resources/__init__.py", line 981, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/third_party/setuptools-36.6.0/pkg_resources/__init__.py", line 867, in resolve
    raise DistributionNotFound(req, requirers)
DistributionNotFound: The 'google-api-core' distribution was not found and is required by the application"

@crwilcox

@crwilcox There's a note in the old google-cloud-python repo that says says the google-cloud-* libs will not work in Py 2 App Engine. https://github.com/googleapis/google-cloud-python#google-cloud-python-client

Heads up! These libraries are supported on App Engine standard's Python 3 runtime but are not supported on App Engine's Python 2 runtime.

Is that use case something we're looking to support?

Support is a strong word. That said, seeing that this dependency is causing issues in both areas makes me feel we may want to see if we can avoid it. The interesting thing is that the issue alpine building is the same.

All that in mind, even if I fix this (right now I mocked it out in vendored versions by just saying the version was a static string, we have issues importing rsa in GAE.

"
Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 238, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 309, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/4a2384f6b69cef45/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 83, in LoadObject
    obj = __import__(path[0])
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/main.py", line 17, in <module>
    from google.cloud import datastore
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/cloud/datastore/__init__.py", line 62, in <module>
    from google.cloud.datastore.batch import Batch
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/cloud/datastore/batch.py", line 24, in <module>
    from google.cloud.datastore import helpers
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/cloud/datastore/helpers.py", line 29, in <module>
    from google.cloud.datastore_v1.proto import datastore_pb2
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/cloud/datastore_v1/__init__.py", line 18, in <module>
    from google.cloud.datastore_v1.gapic import datastore_client
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/cloud/datastore_v1/gapic/datastore_client.py", line 22, in <module>
    from google.oauth2 import service_account
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/oauth2/service_account.py", line 77, in <module>
    from google.auth import _service_account_info
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/auth/_service_account_info.py", line 22, in <module>
    from google.auth import crypt
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/auth/crypt/__init__.py", line 43, in <module>
    from google.auth.crypt import rsa
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/auth/crypt/rsa.py", line 27, in <module>
    from google.auth.crypt import _python_rsa
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/google/auth/crypt/_python_rsa.py", line 28, in <module>
    import rsa
  File "/base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/rsa/__init__.py", line 1
SyntaxError: Non-ASCII character '\xc3' in file /base/data/home/apps/s~crwilcox-test/20201001t152952.429923926239081090/lib/rsa/__init__.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details"

Note that with Python 3, you can have namespace packages without needing pkgutil.

Unfortunately this package is last in line to drop Python 2 support as most of our packages depend on it. Moreover, we are seeing this issue, at least on GAE, on Python 2 specifically. I believe due to the way packages are vendored.

@PureTryOut

@PureTryOut @merwok I released https://pypi.org/project/google-api-core/1.22.4/ which should no longer require get_distribution. Please let me know if this works for your use now :)

Unfortunately for many of the other libraries the use is a little more complicated as its use is in both manually authored and generated code. If this approach does work and other libraries (google-cloud-datastore for instance?) would need further work and likely some fixing of generated surfaces to address this there.

Via #80.

feat: expose library version at `google.auth.__version` (#683)

Replaced usages of the deprecated pkg_resources

Don't use `pkg_resources.get_distribution(..).version`

Don't use `pkg_resources.get_distribution(..).version` (#256)

Migrate from `pkg_resources` to `importlib.metadata`

Migrate from `pkg_resources` to `importlib.metadata` (#1449)

Fast forward experimental-0.2 (#1464)

feat: expose library version at `google.auth.__version` (#683)

Navigation Menu

Repository navigation

Don't use `pkg_resources.get_distribution(..).version` #27

Environment details

Issue

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Don't use pkg_resources.get_distribution(..).version #27

Description

Environment details

Issue

Activity

crwilcox commented on Oct 1, 2020

merwok commented on Oct 1, 2020

crwilcox commented on Oct 1, 2020

crwilcox commented on Oct 1, 2020

crwilcox commented on Oct 1, 2020

busunkim96 commented on Oct 1, 2020

crwilcox commented on Oct 1, 2020

merwok commented on Oct 2, 2020

crwilcox commented on Oct 2, 2020

crwilcox commented on Oct 5, 2020

tseaver commented on Oct 6, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Don't use `pkg_resources.get_distribution(..).version` #27