Skip to content

Address Sanitizer: libasan dead lock in pthread_create() (test_multiprocessing_fork.test_get() hangs) #89363

Closed
Closed
Address Sanitizer: libasan dead lock in pthread_create() (test_multiprocessing_fork.test_get() hangs)#89363
Labels
3.11only security fixesstdlibStandard Library Python modules in the Lib/ directorytestsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

Description

@terryjreedy
terryjreedy
opened on Sep 15, 2021
BPO 45200
Nosy @terryjreedy, @vstinner, @pablogsal
PRs
  • bpo-45020: Don't test IDLE with frozen module. #28344
  • bpo-45200: Ignore test_multiprocessing_* in ASAN build due to false positives #28492
  • bpo-45200: GHA Address Sanitizer skips 3 slowest tests #30797
    		•
    Files
  • bisect10
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2022-01-22.18:17:15.960>
created_at = <Date 2021-09-15.04:17:40.678>
labels = ['tests', 'type-bug', 'library', '3.11']
title = 'Address Sanitizer: libasan dead lock in pthread_create() (test_multiprocessing_fork.test_get() hangs)'
updated_at = <Date 2022-01-22.18:17:15.960>
user = 'https://github.com/terryjreedy'

    bugs.python.org fields:

    activity = <Date 2022-01-22.18:17:15.960>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-22.18:17:15.960>
closer = 'vstinner'
components = ['Library (Lib)', 'Tests']
creation = <Date 2021-09-15.04:17:40.678>
creator = 'terry.reedy'
dependencies = []
files = ['50284']
hgrepos = []
issue_num = 45200
keywords = ['patch']
message_count = 15.0
messages = ['401808', '402075', '402085', '402094', '402095', '402097', '402148', '402152', '402155', '402157', '402205', '402223', '402323', '411273', '411274']
nosy_count = 3.0
nosy_names = ['terry.reedy', 'vstinner', 'pablogsal']
pr_nums = ['28344', '28492', '30797']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45200'
versions = ['Python 3.11']

    Linked PRs

    Activity

    1. terryjreedy commented on Sep 15, 2021

      @terryjreedy
      terryjreedy
      on Sep 15, 2021
      MemberAuthor

      https://github.com/python/cpython/pull/28344/checks?check_run_id=3605759743
      All tests pass until test_multiprocessing_fork timed out after 25 min. On the rerun: refail with timeout.

      test_get (test.test_multiprocessing_fork.WithProcessesTestQueue) ... Timeout (0:20:00)!
      Thread 0x00007f176a71ebc0 (most recent call first):
      File "/home/runner/work/cpython/cpython/Lib/multiprocessing/synchronize.py", line 261 in wait
      File "/home/runner/work/cpython/cpython/Lib/multiprocessing/synchronize.py", line 349 in wait
      File "/home/runner/work/cpython/cpython/Lib/test/test_multiprocessing.py", line 1001 in test_get
      File "/home/runner/work/cpython/cpython/Lib/unittest/case.py", line 549 in _callTestMethod
      File "/home/runner/work/cpython/cpython/Lib/unittest/case.py", line 593 in run
      File "/home/runner/work/cpython/cpython/Lib/unittest/case.py", line 652 in __call      _
      File "/home/runner/work/cpython/cpython/Lib/unittest/suite.py", line 122 in run
      File "/home/runner/work/cpython/cpython/Lib/unittest/suite.py", line 84 in __call__
      File "/home/runner/work/cpython/cpython/Lib/unittest/suite.py", line 122 in run
      File "/home/runner/work/cpython/cpython/Lib/unittest/suite.py", line 84 in __call__
      File "/home/runner/work/cpython/cpython/Lib/unittest/suite.py", line 122 in run
      File "/home/runner/work/cpython/cpython/Lib/unittest/suite.py", line 84 in __call__
      File "/home/runner/work/cpython/cpython/Lib/unittest/runner.py", line 206 in run
      File "/home/runner/work/cpython/cpython/Lib/test/support/init.py", line 998 in _run_suite
      File "/home/runner/work/cpython/cpython/Lib/test/support/init.py", line 1124 in run_unittest
      File "/home/runner/work/cpython/cpython/Lib/test/libregrtest/runtest.py", line 261 in _test_module
      File "/home/runner/work/cpython/cpython/Lib/test/libregrtest/runtest.py", line 297 in _runtest_inner2
      File "/home/runner/work/cpython/cpython/Lib/test/libregrtest/runtest.py", line 340 in _runtest_inner
      File "/home/runner/work/cpython/cpython/Lib/test/libregrtest/runtest.py", line 215 in _runtest
      File "/home/runner/work/cpython/cpython/Lib/test/libregrtest/runtest.py", line 245 in runtest
      File "/home/runner/work/cpython/cpython/Lib/test/libregrtest/main.py", line 337 in rerun_failed_tests
      File "/home/runner/work/cpython/cpython/Lib/test/libregrtest/main.py", line 715 in _main
      File "/home/runner/work/cpython/cpython/Lib/test/libregrtest/main.py", line 658 in main
      File "/home/runner/work/cpython/cpython/Lib/test/libregrtest/main.py", line 736 in main
      File "/home/runner/work/cpython/cpython/Lib/test/main.py", line 2 in <module>
      File "/home/runner/work/cpython/cpython/Lib/runpy.py", line 86 in _run_code
      File "/home/runner/work/cpython/cpython/Lib/runpy.py", line 196 in _run_module_as_main

    2. terryjreedy
      added
      3.11only security fixes
      stdlibStandard Library Python modules in the Lib/ directory
      testsTests in the Lib/test dir
      type-bugAn unexpected behavior, bug, or error
      on Sep 15, 2021

    3. vstinner commented on Sep 17, 2021

      @vstinner
      vstinner
      on Sep 17, 2021
      Member

      test_multiprocessing_fork also failed on the GitHub Action "Address sanitizer" job:
      #28419

      test_get (test.test_multiprocessing_fork.WithProcessesTestQueue) ... Timeout (0:20:00)!
      Thread 0x00007f732679bbc0 (most recent call first):
      File "/home/runner/work/cpython/cpython/Lib/multiprocessing/synchronize.py", line 261 in wait
      File "/home/runner/work/cpython/cpython/Lib/multiprocessing/synchronize.py", line 349 in wait
      File "/home/runner/work/cpython/cpython/Lib/test/_test_multiprocessing.py", line 1001 in test_get

    4. vstinner
      changed the title [-]test_multiprocessing_fork failws with timeout[/-] [+]test_multiprocessing_fork: test_get() fails with timeout[/+] on Sep 17, 2021
    5. vstinner
      changed the title [-]test_multiprocessing_fork failws with timeout[/-] [+]test_multiprocessing_fork: test_get() fails with timeout[/+] on Sep 17, 2021

    6. vstinner commented on Sep 17, 2021

      @vstinner
      vstinner
      on Sep 17, 2021
      Member

      Sometimes, WithProcessesTestProcess.test_error_on_stdio_flush_1() hangs.

    7. vstinner commented on Sep 17, 2021

      @vstinner
      vstinner
      on Sep 17, 2021
      Member

      I'm able to reproduce the issue in a reliable way using attached bisect10 file and the command:

      $ ./python -m test test_multiprocessing_fork --timeout=40 -v --matchfile=bisect10
(...)
test_fork (test.test_multiprocessing_fork.WithProcessesTestQueue) ... ok
test_get (test.test_multiprocessing_fork.WithProcessesTestQueue) ... 
Timeout (0:00:40)!
Thread 0x00007fd0552025c0 (most recent call first):
  File "/home/vstinner/python/main/Lib/multiprocessing/synchronize.py", line 261 in wait
  File "/home/vstinner/python/main/Lib/multiprocessing/synchronize.py", line 349 in wait
  File "/home/vstinner/python/main/Lib/test/_test_multiprocessing.py", line 1003 in test_get
(...)

      bisect10 contains 68 tests.

    8. vstinner commented on Sep 17, 2021

      @vstinner
      vstinner
      on Sep 17, 2021
      Member

      I'm able to reproduce the issue in a reliable way using attached bisect10 file and the command:

      I'm talking about a Python built with Address Sanitizer:

      $ ./configure --with-address-sanitizer --without-pymalloc
$ make

    9. 12 remaining items

    10. vstinner commented on Jan 22, 2022

      @vstinner
      vstinner
      on Jan 22, 2022
      Member

      I reopen the issue: test_concurrent_futures seems to be also affected by this bug :-(

      Example of recent hang:
      https://github.com/python/cpython/runs/4908075699?check_suite_focus=true

      ---
      0:18:04 load avg: 1.34 [417/418] test_peg_generator passed (9 min 46 sec) -- running: test_concurrent_futures (11 min 20 sec)
      0:18:34 load avg: 0.81 running: test_concurrent_futures (11 min 50 sec)
      0:19:04 load avg: 0.49 running: test_concurrent_futures (12 min 20 sec)
      (...)
      0:31:34 load avg: 0.00 running: test_concurrent_futures (24 min 50 sec)
      Kill <TestWorkerProcess #3 running test=test_concurrent_futures pid=20092 time=25 min> process group
      0:31:43 load avg: 0.00 [418/418/1] test_concurrent_futures timed out (25 min) (25 min)
      (...)
      0:31:43 load avg: 0.00 Re-running test_concurrent_futures in verbose mode
      (...)

      OK (skipped=6)

      == Tests result: FAILURE then SUCCESS ==

      409 tests OK.

      10 slowest tests:

      • test_concurrent_futures: 25 min
      • test_peg_generator: 9 min 46 sec
      • test_tools: 4 min 45 sec
      • test_asyncio: 2 min 18 sec
      • test_weakref: 1 min 26 sec
      • test_venv: 1 min 24 sec
      • test_subprocess: 1 min 6 sec
      • test_lib2to3: 1 min 1 sec
      • test_pickle: 59.4 sec
      • test_gdb: 58.5 sec

      9 tests skipped:
      test_devpoll test_ioctl test_kqueue test_msilib test_startfile
      test_winconsoleio test_winreg test_winsound test_zipfile64

      1 re-run test:
      test_concurrent_futures
      ---

    11. vstinner
      reopened this on Jan 22, 2022
    12. vstinner
      reopened this on Jan 22, 2022

    13. vstinner commented on Jan 22, 2022

      @vstinner
      vstinner
      on Jan 22, 2022
      Member

      New changeset ce7d667 by Victor Stinner in branch 'main':
      bpo-45200: GHA Address Sanitizer skips 3 slowest tests (GH-30797)
      ce7d667

    14. vstinner
      closed this as completedon Jan 22, 2022
    15. vstinner
      closed this as completedon Jan 22, 2022
    16. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    17. vstinner
      added a commit that references this issue on Sep 25, 2023

      pythongh-89363: Skip threading test_is_alive_after_fork() if ASAN

      6b2201e
    18. miss-islington
      added a commit that references this issue on Sep 25, 2023

      pythongh-89363: Skip threading test_is_alive_after_fork() if ASAN (py…

      bcb6a79
    19. vstinner
      added a commit that references this issue on Sep 25, 2023

      gh-89363: Skip threading test_is_alive_after_fork() if ASAN (#109835)

      bc06743
    20. miss-islington
      added a commit that references this issue on Sep 25, 2023

      pythongh-89363: Skip threading test_is_alive_after_fork() if ASAN (py…

      42aaca0
    21. vstinner
      added a commit that references this issue on Sep 25, 2023

      [3.11] gh-89363: Skip threading test_is_alive_after_fork() if ASAN (G…

      22a8efa
    22. csm10495
      added a commit that references this issue on Sep 28, 2023

      pythongh-89363: Skip threading test_is_alive_after_fork() if ASAN (py…

      00310f6
    23. Yhg1s
      added a commit that references this issue on Oct 2, 2023

      [3.12] gh-89363: Skip threading test_is_alive_after_fork() if ASAN (G…

      d09a314
    24. Glyphack
      added a commit that references this issue on Sep 2, 2024

      pythongh-89363: Skip threading test_is_alive_after_fork() if ASAN (py…

      51cd05d
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.11only security fixesstdlibStandard Library Python modules in the Lib/ directorytestsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions