Skip to content

Thread sanitizer tests failing in GitHub actions #121946

Description

@colesbury

EDIT: The cause was due to a change in LLVM's Ubuntu package.

I think this is due to a change in GitHub actions, not any particular commit. Commits that previously succeeded are now failing. I cannot reproduce the issue locally.

The only difference I see is in the reported clang build:

CC.version: Ubuntu clang version 18.1.8 (++20240717045746+3b5b5c1ec4a3-1~exp1~20240717165903.140) (bad)
CC.version: Ubuntu clang version 18.1.8 (++20240714071213+3b5b5c1ec4a3-1~exp1~20240714191314.138) (good)

However, the runner IMAGE_VERSION doesn't look like it's changed:

IMAGE_VERSION: 20240714.1.0

Nearly every test is failing:

https://github.com/python/cpython/actions/runs/9982293102/job/27587742397

0:00:00 load avg: 2.83 Run 22 tests in parallel using 4 worker processes
0:00:01 load avg: 2.93 [ 1/22/1] test_capi.test_pyatomic worker non-zero exit code (Exit code 66)
0:00:03 load avg: 2.93 [ 2/22/2] test_code worker non-zero exit code (Exit code 66)
0:00:06 load avg: 3.01 [ 3/22/3] test_functools worker non-zero exit code (Exit code 66)
0:00:11 load avg: 3.01 [ 4/22/4] test_capi.test_mem worker non-zero exit code (Exit code 66)
0:00:12 load avg: 3.01 [ 5/22/5] test_httpservers worker non-zero exit code (Exit code 66)
0:00:16 load avg: 3.01 [ 6/22/6] test_enum worker non-zero exit code (Exit code 66)
0:00:21 load avg: 3.01 [ 7/22/7] test_imaplib worker non-zero exit code (Exit code 66)
0:00:39 load avg: 3.83 [ 8/22/8] test_importlib worker non-zero exit code (Exit code 66)
0:00:41 load avg: 4.01 [ 9/22/9] test_logging worker non-zero exit code (Exit code 66)
0:01:03 load avg: 3.79 [10/22/10] test_io worker non-zero exit code (Exit code 66) -- running (1): test_queue (42.2 sec)
0:01:08 load avg: 4.21 [11/22/11] test_signal worker non-zero exit code (Exit code 66) -- running (1): test_queue (47.1 sec)
0:01:09 load avg: 4.21 [12/22/12] test_sqlite3 worker non-zero exit code (Exit code 66) -- running (1): test_queue (48.0 sec)
0:01:11 load avg: 4.11 [13/22/13] test_syslog worker non-zero exit code (Exit code 66) -- running (2): test_queue (50.3 sec), test_socket (30.1 sec)
0:01:15 load avg: 4.82 [14/22/14] test_thread worker non-zero exit code (Exit code 66) -- running (2): test_queue (54.4 sec), test_socket (34.3 sec)
0:01:17 load avg: 4.82 [15/22/15] test_threadedtempfile worker non-zero exit code (Exit code 66) -- running (2): test_queue (56.2 sec), test_socket (36.1 sec)
0:01:20 load avg: 4.82 [16/22/16] test_queue worker non-zero exit code (Exit code 66) -- running (1): test_socket (39.0 sec)
0:01:23 load avg: 4.68 [17/22/17] test_threading_local worker non-zero exit code (Exit code 66) -- running (1): test_socket (41.8 sec)
0:01:24 load avg: 4.68 [18/22/18] test_ssl worker non-zero exit code (Exit code 66) -- running (1): test_socket (43.3 sec)
0:01:29 load avg: 4.54 [19/22/19] test_threadsignals worker non-zero exit code (Exit code 66) -- running (1): test_socket (48.1 sec)
0:01:50 load avg: 3.91 [20/22/20] test_weakref worker non-zero exit code (Exit code 66) -- running (2): test_threading (32.6 sec), test_socket (1 min 8 sec)
0:01:53 load avg: 3.68 [21/22/21] test_socket worker non-zero exit code (Exit code 66) -- running (1): test_threading (35.4 sec)
0:01:56 load avg: 3.46 [22/22/22] test_threading worker non-zero exit code (Exit code 66)

TSan logs are useless:

WARNING: ThreadSanitizer: data race (pid=8535)
  Write of size 8 at 0x56136380acd0 by main thread:
    #0 <null> <null> (python+0x31a5ee) (BuildId: 243a59[16](https://github.com/python/cpython/actions/runs/9982293102/job/27587742397#step:13:17)b2751fda530fabbdb48342727b53c0b8)
    #1 <null> <null> (python+0x31a4a6) (BuildId: 243a5916b2751fda530fabbdb48342727b53c0b8)
    #2 <null> <null> (python+0x4f1242) (BuildId: 243a5916b2751fda530fabbdb48342727b53c0b8)
    #3 <null> <null> (python+0x522ef7) (BuildId: 243a5916b2751fda530fabbdb48342727b53c0b8)
...

Linked PRs

Activity

  1. added
    testsTests in the Lib/test dir
    buildThe build process and cross-build
    on Jul 18, 2024
  2. vstinner commented on Jul 18, 2024

    @vstinner
    Member

    Can we make the two TSan jobs as non-mandatory until the problem is solved? Right now, it's no longer possible to merge any change.

    cc @ambv

  3. colesbury commented on Jul 18, 2024

    @colesbury
    ContributorAuthor

    I'm looking at this now

  4. colesbury commented on Jul 18, 2024

    @colesbury
    ContributorAuthor

    Ok, it looks like the llvm.sh script from https://apt.llvm.org/ no longer installs the llvm-18 (or llvm-18-dev) package. That means that llvm-symbolizer is missing, which apparently means that:

    1. The stack traces are garbage
    2. The suppressions don't work
  5. colesbury commented on Jul 18, 2024

    @colesbury
    ContributorAuthor

    Looks like this is related: llvm/llvm-project#99502

  6. added 4 commits that reference this issue on Jul 18, 2024
  7. added a commit that references this issue on Jul 18, 2024
  8. added a commit that references this issue on Jul 18, 2024
  9. brandtbucher commented on Jul 18, 2024

    @brandtbucher
    Member

    I believe this is affecting JIT builds as well (llvm.sh isn't installing llvm-readobj). Downgrading is more painful there, so hopefully it's resolved soon.

  10. added 2 commits that reference this issue on Jul 19, 2024
  11. 8 remaining items

  12. added a commit that references this issue on Jul 30, 2024
  13. added a commit that references this issue on Jul 30, 2024
  14. added a commit that references this issue on Jul 30, 2024
  15. added a commit that references this issue on Jul 30, 2024
  16. mhsmith commented on Jul 30, 2024

    @mhsmith
    Member
  17. added a commit that references this issue on Aug 22, 2024
  18. added a commit that references this issue on Apr 14, 2026
  19. added a commit that references this issue on Apr 19, 2026
  20. added 2 commits that reference this issue on Apr 19, 2026
  21. added 2 commits that reference this issue on Apr 19, 2026
  22. hugovk commented on Apr 19, 2026

    @hugovk
    Member

    We're now using clang-20 so I think we can close this.

  23. added a commit that references this issue on Apr 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    buildThe build process and cross-buildtestsTests in the Lib/test dir

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions