Skip to content

mimalloc should fail to allocate 784 271 641 GiB: test_decimal killed by the Linux kernel with OOM on a Free Threading build #114331

Closed
Closed
mimalloc should fail to allocate 784 271 641 GiB: test_decimal killed by the Linux kernel with OOM on a Free Threading build#114331
Labels
topic-free-threadingtype-bugAn unexpected behavior, bug, or error

Description

@hroncok
hroncok
opened on Jan 19, 2024

Bug report

Bug description:

Hello. When we were updating Python 3.13 from a2 to a3 in Fedora, I wanted to enable the freethreding on s390x and ppc64le, as it was previously disabled due to #112535

However, the freethreding build fails to build in Fedora Linux 40 on s390x with:

...
# Next, run the profile task to generate the profile information.
LD_LIBRARY_PATH=/builddir/build/BUILD/Python-3.13.0a3/build/freethreading ./python -m test --pgo --timeout=
Using random seed: 1705622400
0:00:00 load avg: 2.56 Run 44 tests sequentially
0:00:00 load avg: 2.56 [ 1/44] test_array
0:00:01 load avg: 2.56 [ 2/44] test_base64
0:00:02 load avg: 2.56 [ 3/44] test_binascii
0:00:02 load avg: 2.56 [ 4/44] test_binop
0:00:02 load avg: 2.56 [ 5/44] test_bisect
0:00:02 load avg: 2.56 [ 6/44] test_bytes
0:00:09 load avg: 2.63 [ 7/44] test_bz2
0:00:10 load avg: 2.63 [ 8/44] test_cmath
0:00:11 load avg: 2.63 [ 9/44] test_codecs
0:00:13 load avg: 2.63 [10/44] test_collections
0:00:14 load avg: 2.58 [11/44] test_complex
0:00:15 load avg: 2.58 [12/44] test_dataclasses
0:00:15 load avg: 2.58 [13/44] test_datetime
0:00:22 load avg: 2.78 [14/44] test_decimal
make: *** [Makefile:845: profile-run-stamp] Killed

Full log: build.log

I'll try to build this on older Fedora Linux versions as well to eliminate other factors, such as GCC 14:

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

Activity

  1. hroncok
    added
    type-bugAn unexpected behavior, bug, or error
    on Jan 19, 2024

  2. hroncok commented on Jan 19, 2024

    @hroncok
    hroncok
    on Jan 19, 2024
    ContributorAuthor

    Same failures on all supported Fedora versions.

  3. hroncok commented on Jan 20, 2024

    @hroncok
    hroncok
    on Jan 20, 2024
    ContributorAuthor

    @vstinner Is there a buildbot that runs this combination? s390x+freethreading+pgo?

  4. itamaro
    added
    topic-free-threading
    on Jan 20, 2024

  5. corona10 commented on Jan 20, 2024

    @corona10
    corona10
    on Jan 20, 2024
    Member

    It can be gcc bug, if we test with inline assembly instead, suspection will be proved.

  6. corona10 commented on Jan 20, 2024

    @corona10
    corona10
    on Jan 20, 2024
    Member

    @hroncok Can you test with clang too?

  7. hroncok commented on Jan 20, 2024

    @hroncok
    hroncok
    on Jan 20, 2024
    ContributorAuthor

    I got the same failure with clang: build.log

  8. vstinner commented on Jan 22, 2024

    @vstinner
    vstinner
    on Jan 22, 2024
    Member

    @vstinner Is there a buildbot that runs this combination? s390x+freethreading+pgo?

    No. So far, nobody proposed adding a s390x builder for the Free Threading build.

  9. vstinner commented on Jan 22, 2024

    @vstinner
    vstinner
    on Jan 22, 2024
    Member

    mialloc is fine with allocating 842105263157894744 bytes, whereas glibc malloc() fails with NULL. But when Python tries to use this memory (write data), it's killed by the Linux kernel with Out Of Memory (OOM).

    842105263157894744 bytes is around 784 271 641 GiB: it's big. I tested on a machine which only has 8 GiB of memory: so clearly, 784 271 641 GiB cannot be allocated.

    The difference between a regular build and a --disable-gil build is that --disable-gil implies to use mimalloc allocator.

    # ./python -m test.pythoninfo|grep mem
pymem.allocator: mimalloc_debug

    ./python -m test -v test_decimal -m test_maxcontext_exact_arith program is KILLED. Running it in gdb doesn't help, since it's killed by the Linux kernel.

    $ journalctl -k -r

Jan 22 09:02:48 s390x-kvm-112.lab.eng.rdu2.redhat.com kernel: Out of memory: Killed process 1549896 (python) total-vm:822368422334244kB, anon-rss:7572064kB, file-rss:8kB, shmem-rss:0kB, UID:0 pgtables:30506kB oom_score_adj:0

    "total-vm:822368422334244kB" that's a lot of memory.

    I put a breakpoint on PyMem_Malloc() if size is larger than 1 GiB:

    Breakpoint 1, PyMem_Malloc (size=842105263157894744) at Objects/obmalloc.c:861
861	    if (size > (size_t)PY_SSIZE_T_MAX)

(gdb) where
#0  PyMem_Malloc (size=842105263157894744) at Objects/obmalloc.c:861
#1  0x000003fff00a1fe2 in mpd_alloc (nmemb=nmemb@entry=105263157894736843, size=size@entry=8) at ./Modules/_decimal/libmpdec/mpalloc.c:92
#2  0x000003fff00a21e6 in mpd_switch_to_dyn (result=result@entry=0x3ffffffa360, nwords=nwords@entry=105263157894736843, status=status@entry=0x3ffffffa6b4) at ./Modules/_decimal/libmpdec/mpalloc.c:217
#3  0x000003fff00a68c0 in mpd_qresize (result=result@entry=0x3ffffffa360, nwords=nwords@entry=105263157894736843, status=status@entry=0x3ffffffa6b4) at ./Modules/_decimal/libmpdec/mpdecimal.c:517
#4  0x000003fff00a99f4 in mpd_qshiftl (result=result@entry=0x3ffffffa360, a=a@entry=0x3ffffffa360, n=n@entry=1999999999999999998, status=status@entry=0x3ffffffa6b4) at ./Modules/_decimal/libmpdec/mpdecimal.c:2520
#5  0x000003fff00b516c in _mpd_qsqrt (result=result@entry=0x200012eb8c8, a=a@entry=0x200012eb808, ctx=ctx@entry=0x20001587520, status=status@entry=0x3ffffffa6b4) at ./Modules/_decimal/libmpdec/mpdecimal.c:7951
#6  0x000003fff00b57d0 in mpd_qsqrt (result=result@entry=0x200012eb8c8, a=a@entry=0x200012eb808, ctx=0x20001587520, status=status@entry=0x3ffffffa98c) at ./Modules/_decimal/libmpdec/mpdecimal.c:8043
#7  0x000003fff008f7e6 in dec_mpd_qsqrt (self=0x200012eb7e0, args=<optimized out>, kwds=<optimized out>) at ./Modules/_decimal/_decimal.c:4407

(gdb) py-bt
Traceback (most recent call first):
  File "/root/cpython/Lib/test/test_decimal.py", line 5665, in test_maxcontext_exact_arith
    self.assertEqual(Decimal(4).sqrt(), 2)

    The test is decorated with:

        @unittest.skipIf(sys.platform.startswith("aix"),
                     "AIX: default ulimit: test is flaky because of extreme over-allocation")
    @unittest.skipIf(is_emscripten, "Test is unstable on Emscripten")
    @unittest.skipIf(check_sanitizer(address=True, memory=True),
                     "ASAN/MSAN sanitizer defaults to crashing "
                     "instead of returning NULL for malloc failure.")

    On a regular Python build, I see the same memory allocation:

    0:00:00 load avg: 0.84 [1/1] test_decimal
test_maxcontext_exact_arith (test.test_decimal.CWhitebox.test_maxcontext_exact_arith) ... 
Breakpoint 1, PyMem_Malloc (size=842105263157894744) at Objects/obmalloc.c:861
861	    if (size > (size_t)PY_SSIZE_T_MAX)

(gdb) py-bt
Traceback (most recent call first):
  File "/root/cpython/Lib/test/test_decimal.py", line 5665, in test_maxcontext_exact_arith
    self.assertEqual(Decimal(4).sqrt(), 2)

    There are 4 large memory allocations, all of them fail:

    • size=842105263157894744
    • size=842105263157894744
    • size=421052631578947376
    • size=421052631578947376
  10. vstinner
    changed the title [-]3.13.0a3 freethreding build fails to build on s390x: killed during the profile task[/-] [+]mimalloc should to allocate 784 271 641 GiB: test_decimal killed by the Linux kernel with OOM on a Free Threading build[/+] on Jan 22, 2024

  11. vstinner commented on Jan 22, 2024

    @vstinner
    vstinner
    on Jan 22, 2024
    Member

    @DinoV @colesbury: mimalloc needs maybe some tuning here?

  12. vstinner commented on Jan 22, 2024

    @vstinner
    vstinner
    on Jan 22, 2024
    Member

    --disable-gil on x86-64 seems to behave differently. On my laptop with 32 GiB of RAM:

    0:00:00 load avg: 1.01 [1/1] test_decimal
test_maxcontext_exact_arith (test.test_decimal.CWhitebox.test_maxcontext_exact_arith) ... 
Breakpoint 1, PyMem_Malloc (size=842105263157894744) at Objects/obmalloc.c:861
861	    if (size > (size_t)PY_SSIZE_T_MAX)
Missing separate debuginfos, use: dnf debuginfo-install bzip2-libs-1.0.8-16.fc39.x86_64 xz-libs-5.4.4-1.fc39.x86_64 zlib-1.2.13-4.fc39.x86_64
(gdb) n
867	    return _PyMem.malloc(_PyMem.ctx, size);
(gdb) n

mimalloc: warning: unable to allocate OS memory (error: 12 (0xc), size: 0xbafc24672800000 bytes, align: 0x2000000, commit: 1, allow large: 1)
mimalloc: warning: unable to allocate OS memory (error: 12 (0xc), size: 0xbafc24672800000 bytes, align: 0x2000000, commit: 1, allow large: 1)
mimalloc: error: unable to allocate memory (842105263157894768 bytes)

868	}
(gdb) n
mpd_alloc (nmemb=105263157894736843, size=8) at ./Modules/_decimal/libmpdec/mpalloc.c:93
93	}
(gdb) n
mpd_switch_to_dyn (result=0x7ffffffe22c0, nwords=105263157894736843, status=0x7ffffffe25cc) at ./Modules/_decimal/libmpdec/mpalloc.c:218
218	    if (result->data == NULL) {
(gdb) p result->data
$1 = (mpd_uint_t *) 0x0

    See "warning: unable to allocate OS memory" error.

    Using strace, I see mmap() syscalls failing with ENOMEM.

    mmap(NULL, 842105263166062592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire)
write(2, "mimalloc: warning: ", 19)     = 19
write(2, "unable to allocate OS memory (er"..., 123) = 123
mmap(NULL, 842105263166062592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire)
write(2, "mimalloc: warning: ", 19)     = 19
write(2, "unable to allocate OS memory (er"..., 123) = 123
write(2, "mimalloc: error: ", 17)       = 17
write(2, "unable to allocate memory (84210"..., 53) = 53
mmap(NULL, 842105263166062592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire)
write(2, "mimalloc: warning: ", 19)     = 19
write(2, "unable to allocate OS memory (er"..., 123) = 123
mmap(NULL, 842105263166062592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire)
write(2, "mimalloc: warning: ", 19)     = 19
write(2, "unable to allocate OS memory (er"..., 123) = 123
write(2, "mimalloc: error: ", 17)       = 17
write(2, "unable to allocate memory (84210"..., 53) = 53
mmap(NULL, 421052631587225600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire)
write(2, "mimalloc: warning: ", 19)     = 19
write(2, "unable to allocate OS memory (er"..., 123) = 123
mmap(NULL, 421052631587225600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire)
write(2, "mimalloc: warning: ", 19)     = 19
write(2, "unable to allocate OS memory (er"..., 123) = 123
write(2, "mimalloc: error: ", 17)       = 17
write(2, "unable to allocate memory (42105"..., 53) = 53
mmap(NULL, 421052631587225600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire)
write(2, "mimalloc: warning: ", 19)     = 19
write(2, "unable to allocate OS memory (er"..., 123) = 123
mmap(NULL, 421052631587225600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Ne peut allouer de la mémoire)
write(2, "mimalloc: warning: ", 19)     = 19
write(2, "unable to allocate OS memory (er"..., 123) = 123
write(2, "mimalloc: error: ", 17)       = 17
write(2, "unable to allocate memory (42105"..., 53) = 53

  13. vstinner commented on Jan 22, 2024

    @vstinner
    vstinner
    on Jan 22, 2024
    Member

    @DinoV @colesbury: Tell me if you need more information about s390x.

  14. colesbury commented on Jan 22, 2024

    @colesbury
    colesbury
    on Jan 22, 2024
    Contributor

    Thanks for debugging this and find the rooting cause. My inclination is to address this in _decimal.c (or possibly test_decimal.py) rather than trying to add heuristics to mimalloc. This makes the fourth platform/configuration with problems with this test -- the behavior it assumes does not seem reliable.

    In the meantime, if we want to build/test a free-threaded build on s390x, we can add another unittest.skipIf to the test.

  15. vstinner commented on Jan 22, 2024

    @vstinner
    vstinner
    on Jan 22, 2024
    Member

    I would be fine with a skip on s390x, since apparently, the underlying memory allocator or Linux kernel behave differently on s390x and x86-64.

  16. 7 remaining items

  17. vstinner commented on Mar 28, 2024

    @vstinner
    vstinner
    on Mar 28, 2024
    Member

    I proposed PR gh-117326 to skip test_decimal.test_maxcontext_exact_arith() on s390x.

  18. vstinner
    added a commit that references this issue on Mar 28, 2024

    gh-114331: Skip decimal test_maxcontext_exact_arith on s390x (#117326)

    6702d2b
  19. miss-islington
    added 2 commits that reference this issue on Mar 28, 2024

    pythongh-114331: Skip decimal test_maxcontext_exact_arith on s390x (p…

    3f6b65a

    pythongh-114331: Skip decimal test_maxcontext_exact_arith on s390x (p…

    74a9df2
  20. vstinner
    added a commit that references this issue on Mar 28, 2024

    [3.12] gh-114331: Skip decimal test_maxcontext_exact_arith on s390x (G…

    1e48de6

  21. vstinner commented on Mar 28, 2024

    @vstinner
    vstinner
    on Mar 28, 2024
    Member

    Issue fixed by change 6702d2b.

  22. vstinner
    closed this as completedon Mar 28, 2024
  23. vstinner
    added a commit that references this issue on Mar 28, 2024

    [3.11] gh-114331: Skip decimal test_maxcontext_exact_arith on s390x (#…

    8bbb121

  24. colesbury commented on Mar 28, 2024

    @colesbury
    colesbury
    on Mar 28, 2024
    Contributor

    Thanks Victor!

  25. NGRsoftlab
    added 2 commits that reference this issue on Apr 2, 2024

    Squashed commit of the following:

    1b5fe4c

    Squashed commit of the following:

    a016a54
  26. vstinner
    added 2 commits that reference this issue on Apr 16, 2024

    pythongh-117755: Remove tests on huge memory allocations

    cf45a92

    gh-117755: Remove tests on huge memory allocations (#117938)

    e05d202
  27. diegorusso
    added 2 commits that reference this issue on Apr 17, 2024

    pythongh-114331: Skip decimal test_maxcontext_exact_arith on s390x (p…

    e112420

    pythongh-117755: Remove tests on huge memory allocations (python#117938)

    80cd6a2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic-free-threadingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions