Skip to content

test_venv failure when executed by test_multiprocessing and the platform lacks a functional sem_open() #80523

Closed
Closed
test_venv failure when executed by test_multiprocessing and the platform lacks a functional sem_open()#80523
Labels
3.8 (EOL)end of lifetestsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

Description

@xdegaye
xdegaye
mannequin
opened on Mar 18, 2019
BPO 36342
Nosy @brettcannon, @vstinner, @xdegaye, @miss-islington
PRs
  • bpo-36342: Fix test_multiprocessing in test_venv #12513
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-09-24.12:06:09.204>
created_at = <Date 2019-03-18.10:02:39.890>
labels = ['3.8', 'type-bug', 'tests']
title = 'test_venv failure when executed by test_multiprocessing and the platform lacks a functional sem_open()'
updated_at = <Date 2019-09-24.12:06:09.203>
user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2019-09-24.12:06:09.203>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-24.12:06:09.204>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2019-03-18.10:02:39.890>
creator = 'xdegaye'
dependencies = []
files = []
hgrepos = []
issue_num = 36342
keywords = ['patch']
message_count = 10.0
messages = ['338188', '338189', '338192', '338240', '338398', '338464', '338495', '338671', '344004', '353078']
nosy_count = 4.0
nosy_names = ['brett.cannon', 'vstinner', 'xdegaye', 'miss-islington']
pr_nums = ['12513']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36342'
versions = ['Python 3.8']

    Activity

    1. xdegaye commented on Mar 18, 2019

      @xdegaye
      xdegayemannequin
      on Mar 18, 2019
      MannequinAuthor

      test_multiprocessing fails on platforms lacking a functioning sem_open (Android for example) with:

      ======================================================================
      ERROR: test_multiprocessing (test.test_venv.BasicTest)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.8/test/test_venv.py", line 317, in test_multiprocessing
    out, err = check_output([envpy, '-c',
  File "/data/local/tmp/python/lib/python3.8/test/test_venv.py", line 37, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['/data/local/tmp/python/tmp/tmpli0_hkdl/bin/python', '-c', 'from multiprocessing import Pool; print(Pool(1).apply_async("Python".lower).get(3))']' returned non-zero exit status 1.
    2. xdegaye
      added
      3.8 (EOL)end of life
      testsTests in the Lib/test dir
      type-bugAn unexpected behavior, bug, or error
      on Mar 18, 2019

    3. vstinner commented on Mar 18, 2019

      @vstinner
      vstinner
      on Mar 18, 2019
      Member

      Are you sure that multiprocessing is the issue? On bpo-35978, the test fails because of venv, not because of multiprocessing. Can it be the same issue?

    4. xdegaye commented on Mar 18, 2019

      @xdegaye
      xdegayemannequin
      on Mar 18, 2019
      MannequinAuthor

      You are right, this is a duplicate of bpo-35978.
      Closing as duplicate.

    5. xdegaye
      closed this as completedon Mar 18, 2019
    6. xdegaye
      changed the title [-]test_multiprocessing fails on platforms lacking a functioning sem_open[/-] [+]test_venv fails on Android with clang[/+] on Mar 18, 2019

    7. xdegaye commented on Mar 18, 2019

      @xdegaye
      xdegayemannequin
      on Mar 18, 2019
      MannequinAuthor

      Too hasty again, remembering now why I linked this failure with sem_open :-(

      Re-opening this issue as the test does fail because Android lacks a functioning sem_open implementation. And tagging this issue as a dependency of bpo-35978 as we should wait for the solution on bpo-35978 before fixing this one, if it is still needed.

      The link between the failure and sem_open shows up when running the python statements of test_multiprocessing() in the interpreter on Android:

      generic_x86_64:/data/local/tmp/python $ python
Python 3.8.0a2+ (heads/bpo-36341-dirty:41f0b78cbf, Mar 18 2019, 10:43:05)
[Clang 3.8.275480 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> print(Pool(1).apply_async("Python".lower).get(3))
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/local/tmp/python/lib/python3.8/lib-dynload/_multiprocessing.cpython-38dm.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/context.py", line 118, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/pool.py", line 166, in __init__
    self._setup_queues()
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/pool.py", line 300, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/context.py", line 112, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/context.py", line 66, in Lock
    from .synchronize import Lock
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
>>>
    8. xdegaye
      reopened this on Mar 18, 2019

    9. brettcannon commented on Mar 19, 2019

      @brettcannon
      brettcannon
      on Mar 19, 2019
      Member

      So this is a test_multiprocessing issue? If it is then the title is misleading.

    10. xdegaye commented on Mar 20, 2019

      @xdegaye
      xdegayemannequin
      on Mar 20, 2019
      MannequinAuthor

      It is a test_venv issue related to bpo-32126 and also related to the other issues listed in bpo-32126.

      Changing the title as it is ambiguous indeed.

      check_output() in test_venv.py does not print the full stack trace as it is done by test_asyncio in bpo-32126, is this another issue ?

    11. xdegaye
      changed the title [-]test_venv fails on Android with clang[/-] [+]test_venv failure when the platform lacks a functional sem_open()[/+] on Mar 20, 2019

    12. brettcannon commented on Mar 20, 2019

      @brettcannon
      brettcannon
      on Mar 20, 2019
      Member

      I guess my confusion comes from the fact that test_venv in isolation is a totally fine test suite, it just fails when test_multiprocessing runs it due to something multiprocessing-related.

      I've tried tweaking the title to reflect the fact that it's a test_venv failure triggered by test_multiprocessing.

    13. brettcannon
      changed the title [-]test_venv failure when the platform lacks a functional sem_open()[/-] [+]test_venv failure when executed by test_multiprocessing and the platform lacks a functional sem_open()[/+] on Mar 20, 2019

    14. xdegaye commented on Mar 23, 2019

      @xdegaye
      xdegayemannequin
      on Mar 23, 2019
      MannequinAuthor

      Removing the bpo-35978 dependency as changeset 8bba81f that fixes bpo-35978 only skips test_multiprocessing when test_venv is run from a venv.

    15. miss-islington commented on May 30, 2019

      @miss-islington
      miss-islington
      on May 30, 2019
      Contributor

      New changeset 5437ccc by Miss Islington (bot) (xdegaye) in branch 'master':
      bpo-36342: Fix test_multiprocessing in test_venv (GH-12513)
      5437ccc

    16. vstinner commented on Sep 24, 2019

      @vstinner
      vstinner
      on Sep 24, 2019
      Member

      It seems like this issue has been fixed in the master branch. Reopen the issue if you consider that the fix should be backported.

    17. vstinner
      closed this as completedon Sep 24, 2019
    18. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.8 (EOL)end of lifetestsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions