Repository navigation
SIGSEV in datetime.timedelta (possibly from datetime's C delta_new) #132413
Description
Activity
I was testing some AI code with ollama and stumbled across a really weird crash.
What's the crash? please provide the traceback (shown on the terminal if possible) (not just the core dump).
Sorry. Here is the python traceback. If you wanted a gdb or other traceback, let me know.
File "crasher1.py", line 19, in <module> temp[1] ~~~~^^^ IndexError: list index out of range Segmentation fault (core dumped)If you wanted a gdb or other traceback, let me know
If possible yes, so that we know where the crash exactly happens. If possible, you can use
python -X faulthandler crasher.pyas well though I'm not sure if we're able to know more.Also, it might be that
ollamais using the Python C API behind the scene (AFAIU,ollamais written in Go and C but there are Python bindings, which may be the ones where the issue arise).- changed the title
[-]Segfault on IndexError under specific conditions[/-][+]SIGSEV following an `IndexError` when using `ollama`[/+]on Apr 11, 2025 Ah. It appears that it is an ollama issue after all. Ill raise it over there.
Fatal Python error: Segmentation fault Current thread 0x00007d584222bbc0 (most recent call first): Garbage-collecting File "/home/jacoblightning3/PycharmProjects/AiFight/.venv/lib/python3.13/site-packages/httpx/_client.py", line 158 in close File "/home/jacoblightning3/PycharmProjects/AiFight/.venv/lib/python3.13/site-packages/httpx/_models.py", line 972 in close File "/home/jacoblightning3/PycharmProjects/AiFight/.venv/lib/python3.13/site-packages/httpx/_client.py", line 877 in stream File "/usr/lib/python3.13/contextlib.py", line 162 in __exit__ File "/home/jacoblightning3/PycharmProjects/AiFight/.venv/lib/python3.13/site-packages/ollama/_client.py", line 163 in inner Segmentation fault (core dumped)Reacted by Bénédikt Tran@picnixz So, I just checked and it appears that ollama-python is pure python. (Ofc I realized this after I made the issue.) Would that bring the issue back here? They don't appear to be using CTypes, etc.
That does look like it could be a CPython issue, though ollama has a few dependencies that include compiled code. I tried to reproduce it (3.13.1, MacOS, latest ollama) but I got
httpx.ConnectError: [Errno 61] Connection refusedinstead (does ollama require a local server or something? probably not something I'm interested in setting up).Two useful ways forward could be:
- Get the full C stack trace in gdb or lldb or a similar tool and explore what's happening when we hit the segfault. For example, maybe that will tell you what type it's looking at when the crash happens.
- Reduce the reproducer to something simpler. For example, you can start by removing more and more parts of ollama that aren't relevant to the crash and then see if it still reproduces.
I tried to reproduce it (3.13.1, MacOS, latest ollama) but I got
httpx.ConnectError: [Errno 61] Connection refusedinstead (does ollama require a local server or something? probably not something I'm interested in setting up).Yes. The ollama package requires ollama to be installed and running a local server.
Running python with the debug build I just compiled produces:
Modules/_datetimemodule.c:2745:13: runtime error: member access within null pointer of type 'struct datetime_state' AddressSanitizer:DEADLYSIGNAL ================================================================= ==9445==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000010 (pc 0x7c8cc6b93737 bp 0x7ffe5c2d11f0 sp 0x7ffe5c2d1040 T0) ==9445==The signal is caused by a READ memory access. ==9445==Hint: address points to the zero page. #0 0x7c8cc6b93737 in delta_new Modules/_datetimemodule.c:2745 #1 0x5790c86108b1 in type_call Objects/typeobject.c:1987 #2 0x5790c8410b28 in _PyObject_MakeTpCall Objects/call.c:242 #3 0x5790c8410fc7 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:166 #4 0x5790c8411018 in PyObject_Vectorcall Objects/call.c:327 #5 0x5790c883c457 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1502 #6 0x5790c8472fda in _PyEval_EvalFrame Include/internal/pycore_ceval.h:119 #7 0x5790c8473ce6 in gen_send_ex2 Objects/genobject.c:229 #8 0x5790c84786a3 in gen_send_ex Objects/genobject.c:270 #9 0x5790c847adb5 in _gen_throw Objects/genobject.c:543 #10 0x5790c847b17c in gen_throw Objects/genobject.c:580 #11 0x5790c883ebd2 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1640 #12 0x5790c8883c6b in _PyEval_EvalFrame Include/internal/pycore_ceval.h:119 #13 0x5790c8883f2f in _PyEval_Vector Python/ceval.c:1812 #14 0x5790c84105c3 in _PyFunction_Vectorcall Objects/call.c:413 #15 0x5790c8419a0e in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168 #16 0x5790c841b924 in method_vectorcall Objects/classobject.c:62 #17 0x5790c8410ed7 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168 #18 0x5790c8411018 in PyObject_Vectorcall Objects/call.c:327 #19 0x5790c888107e in _PyEval_EvalFrameDefault Python/generated_cases.c.h:6205 #20 0x5790c8472fda in _PyEval_EvalFrame Include/internal/pycore_ceval.h:119 #21 0x5790c8473ce6 in gen_send_ex2 Objects/genobject.c:229 #22 0x5790c84786a3 in gen_send_ex Objects/genobject.c:270 #23 0x5790c8479898 in gen_close Objects/genobject.c:392 #24 0x5790c8479dcd in _PyGen_Finalize Objects/genobject.c:106 #25 0x5790c894557a in finalize_garbage Python/gc.c:980 #26 0x5790c8947b80 in gc_collect_main Python/gc.c:1408 #27 0x5790c8949657 in _PyGC_CollectNoFail Python/gc.c:1657 #28 0x5790c89fc3c1 in finalize_modules Python/pylifecycle.c:1757 #29 0x5790c8a08ab6 in _Py_Finalize Python/pylifecycle.c:2125 #30 0x5790c8a08fbc in Py_FinalizeEx Python/pylifecycle.c:2252 #31 0x5790c8ab41fa in Py_RunMain Modules/main.c:778 #32 0x5790c8ab440a in pymain_main Modules/main.c:806 #33 0x5790c8ab4787 in Py_BytesMain Modules/main.c:830 #34 0x5790c813ab41 in main Programs/python.c:15 #35 0x7c8cc7c35487 (/usr/lib/libc.so.6+0x27487) (BuildId: 0b707b217b15b106c25fe51df3724b25848310c0) #36 0x7c8cc7c3554b in __libc_start_main (/usr/lib/libc.so.6+0x2754b) (BuildId: 0b707b217b15b106c25fe51df3724b25848310c0) #37 0x5790c813aa64 in _start (/home/jacoblightning3/Documents/python/3.13debug/bin/python3.13+0x11a5a64) (BuildId: c2486f8b3c246fe979c4f6575406636585088b8a) AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV Modules/_datetimemodule.c:2745 in delta_new ==9445==ABORTINGOn current 3.13 tip that line in datetime is
which doesn't have an obvious bug. This sort of crash could be the result of memory corruption earlier in the program.cpython/Modules/_datetimemodule.c
Line 2745 in 089c43f
So, using GDB. I figured out that
ston that line is a null pointer and the macro attempts to dereference it.
stis created in the function_get_current_statewhich GDB says is returning NULL from line 168.
The weird thing is that GDB won't let me step into or break on lines 166 or 167.So, assuming that GDB is right on where
_get_current_stateis returning,get_module_statemust have been called.
cpython/Modules/_datetimemodule.c
Lines 102 to 108 in 089c43f
The other weird thing is that the assert on line 106 is not failing as it should (I have assertions turned on in my build)
So I have to assume that GDB is wrong about where_get_current_stateis returning. (So it must be either 157 or 163).Since neither of those return paths in
_get_current_stateset*p_mod, wouldn't this be a bug sincecurrent_modis not checked indelta_newand it is just assumed that st != NULL?(I could be totally wrong on all this. :)
26 remaining items
Generator's different behaviors are still surprising to me:
def gen(): try: print(1) yield 2 finally: print(3)
print(next(it := gen()))
1 2 3print(next(gen()))
1 3 2@neonene JFYI, cannot repro this without generator.
Reacted by neonenecannot repro this without generator.
Neither can I around__del__.Another repro:
import _datetime class C: def __del__(self): _datetime.timedelta(days=1) # crash l = [C()] l.append(l)
Reacted by Sergey MiryanovInteresting, that this code does
not crash(Of course not - I forgot to create C, blame on me!):import _datetime class C: def __init__(self): self.self = self def __del__(self): _datetime.timedelta(days=1) # crash C() # added after editBut this do:
import _datetime class C: def __del__(self): _datetime.timedelta(days=1) # crash c = C() c.c = cBoth crashed.
Reacted by Dominique DERRIERRegarding backporting, our inclination right now is to be cautious about backporting this. Maybe it should be backported to 3.14 but not to 3.13? AFAIK, we only have two known bugs caused by this, gh-137562 (which causes segfaults do to stale type caches) and this one.
The former seems pretty difficult to provoke. This one is easier since any del method that uses the datetime module might crash. I would guess it's not so uncommon to do some logging there, for example, and that would want a timestamp.
The reason not to backport is that the change in behavior could cause working programs to fail (any bugfix could but the change in when weakrefs are cleared seems potentially more risky). The reason to backport is that this likely fixes a number of mysterious segfaults. Debugging crashes caused by this is probably really hard.
Perhaps we should let this change cook in the 3.15 branch for a while and when we are more sure it doesn't cause problems re-assess backporting it.
Reacted by neoneneFWIW we saw a similar
datetimerelated crash during garbage collection as a 3.13.9 process was exiting today:Current thread 0x... (most recent call first): -- Garbage-collecting File ".../python3.13/site-packages/tqdm/std.py", line 576 in format_meter File ".../python3.13/site-packages/tqdm/std.py", line 1151 in __str__ File ".../python3.13/site-packages/tqdm/std.py", line 1495 in display File ".../python3.13/site-packages/tqdm/std.py", line 1302 in close File ".../python3.13/site-packages/tqdm/std.py", line 1148 in __del__ Current thread's C stack trace (most recent call first): -- Binary file ".../libpython3.13.so.1.0", at _Py_DumpStack+0x4f Binary file ".../libpython3.13.so.1.0", at +0x16d586 Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x42520 Binary file ".../python3.13/lib-dynload/_datetime.cpython-313-x86_64-linux-gnu.so", at +0x9dd4 Binary file ".../libpython3.13.so.1.0", at _PyObject_MakeTpCall+0x211 Binary file ".../libpython3.13.so.1.0", at _PyEval_EvalFrameDefault+0xaa4d Binary file ".../libpython3.13.so.1.0", at PyObject_CallOneArg+0x6anot something readily reproduced. The python gc traceback was in
tqdmversion 4.67.1.https://github.com/tqdm/tqdm/blob/0ed5d7f18fa3153834cbac0aa57e8092b217cc16/tqdm/std.py#L576-L577
eta_dt = (datetime.now() + timedelta(seconds=remaining) if rate and total else datetime.fromtimestamp(0, timezone.utc))
(comment not quite elided about
__del__finalizers that do complicated things like grabbing locks and IO ... 💩)- marked Python 3.13.13 & 3.14.4 segfault crash (10 lines reproducer included) #148732 as a duplicate of this issue
on Apr 19, 2026 I've got two similar crashes with Python 3.14.5 while running
/usr/bin/gmi pull(from lieer).coredumpctl debug
PID: 190349 (gmi) TID: 190349 (gmi) UID: 1000 (marmis) GID: 1000 (marmis) Signal: 11 (SEGV) Timestamp: Thu 2026-06-04 21:02:00 -03 (19s ago) Command Line: /usr/bin/python /usr/bin/gmi pull -C /home/marmis/.local/share/notmuch/tiagodepalves@gmail.com Executable: /usr/bin/python3.14 Control Group: /user.slice/user-1000.slice/user@1000.service/session.slice/kitty-6584-13.scope Unit: user@1000.service User Unit: kitty-6584-13.scope Slice: user-1000.slice Owner UID: 1000 (marmis) Boot ID: 5a0037b704d24555851a501a3ed02a43 Machine ID: 98fc705f46884369b30a5207646ed499 Hostname: marmis-arch Storage: /var/lib/systemd/coredump/core.gmi.1000.5a0037b704d24555851a501a3ed02a43.190349.1780617720000000.zst (present) Size on Disk: 10.3M Message: Process 190349 (gmi) of user 1000 dumped core. Stack trace of thread 190349: #0 0x00007fa01c56dea3 n/a (libpython3.14.so.1.0 + 0x36dea3) #1 0x00007fa01c3a49e8 n/a (libpython3.14.so.1.0 + 0x1a49e8) #2 0x00007fa01c3a48c9 _PyObject_MakeTpCall (libpython3.14.so.1.0 + 0x1a48c9) #3 0x00007fa01c267e21 n/a (libpython3.14.so.1.0 + 0x67e21) #4 0x00007fa01c3e24eb n/a (libpython3.14.so.1.0 + 0x1e24eb) #5 0x00007fa01c3e97d6 PyObject_CallOneArg (libpython3.14.so.1.0 + 0x1e97d6) #6 0x00007fa01c52050c n/a (libpython3.14.so.1.0 + 0x32050c) #7 0x00007fa01c3bd694 n/a (libpython3.14.so.1.0 + 0x1bd694) #8 0x00007fa01c508ed6 n/a (libpython3.14.so.1.0 + 0x308ed6) #9 0x00007fa01c4e109a n/a (libpython3.14.so.1.0 + 0x2e109a) #10 0x00007fa01c4a796a Py_RunMain (libpython3.14.so.1.0 + 0x2a796a) #11 0x00007fa01c4a167c Py_BytesMain (libpython3.14.so.1.0 + 0x2a167c) #12 0x00007fa01be27741 n/a (libc.so.6 + 0x27741) #13 0x00007fa01be27879 __libc_start_main (libc.so.6 + 0x27879) #14 0x00005619d7ec2045 _start (python3.14 + 0x1045) ELF object binary architecture: AMD x86-64 GNU gdb (GDB) 17.2 Copyright (C) 2025 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/python3.14... Reading symbols from /home/marmis/.cache/debuginfod_client/7d28bb88eb2d929bf00b0f7e80bb83480668eae9/debuginfo... [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/libthread_db.so.1". Core was generated by `/usr/bin/gmi /usr/bin/gmi pull -C /home/marmis/.local/share/notmuch/tiagodepalves@gmail.com'. Program terminated with signal SIGSEGV, Segmentation fault. #0 delta_new (type=0x7fa01c7aa380 <PyDateTime_DeltaType.lto_priv.0>, args=<optimized out>, kw=<optimized out>) at ./Modules/_datetimemodule.c:2835 2835 y = accum("seconds", x, second, CONST_US_PER_SECOND(st), &leftover_us); (gdb) bt #0 delta_new (type=0x7fa01c7aa380 <PyDateTime_DeltaType.lto_priv.0>, args=<optimized out>, kw=<optimized out>) at ./Modules/_datetimemodule.c:2835 #1 0x00007fa01c3a49e8 in type_call (self=0x7fa01c7aa380 <PyDateTime_DeltaType.lto_priv.0>, args=0x7fa01c7dd390 <_PyRuntime+90640>, kwds=0x7fa0167a32c0) at Objects/typeobject.c:2372 #2 0x00007fa01c3a48c9 in _PyObject_MakeTpCall (tstate=0x7fa01c8142a0 <_PyRuntime+315680>, callable=0x7fa01c7aa380 <PyDateTime_DeltaType.lto_priv.0>, args=0x7ffc17709c88, nargs=<optimized out>, keywords=0x7fa01596ac00) at Objects/call.c:242 #3 0x00007fa01c267e21 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7fa01c926268, throwflag=<optimized out>) at Python/generated_cases.c.h:3229 #4 0x00007fa01c3e24eb in _PyEval_EvalFrame (throwflag=0, tstate=0x7fa01c8142a0 <_PyRuntime+315680>, frame=0x7fa01c926020) at ./Include/internal/pycore_ceval.h:120 #5 _PyEval_Vector (locals=0x0, tstate=0x7fa01c8142a0 <_PyRuntime+315680>, func=<optimized out>, args=<optimized out>, argcount=1, kwnames=<optimized out>) at Python/ceval.c:2110 #6 _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:413 #7 0x00007fa01c3e97d6 in _PyObject_VectorcallTstate (nargsf=9223372036854775809, kwnames=0x0, tstate=0x7fa01c8142a0 <_PyRuntime+315680>, callable=0x7fa015992980, args=0x7ffc17709e68) at ./Include/internal/pycore_call.h:177 #8 PyObject_CallOneArg (func=0x7fa015992980, arg=<optimized out>) at Objects/call.c:395 #9 0x00007fa01c52050c in call_unbound_noarg (unbound=<optimized out>, func=0x7fa015992980, self=0x7fa0159dfe00) at Objects/typeobject.c:2964 #10 slot_tp_finalize (self=0x7fa0159dfe00) at Objects/typeobject.c:10599 #11 0x00007fa01c3bd694 in finalize_garbage (tstate=0x7fa01c8142a0 <_PyRuntime+315680>, collectable=0x7ffc17709f60) at Python/gc.c:1014 #12 gc_collect_main (tstate=0x7fa01c8142a0 <_PyRuntime+315680>, generation=<optimized out>, reason=_Py_GC_REASON_SHUTDOWN) at Python/gc.c:1446 #13 0x00007fa01c508ed6 in _PyGC_CollectNoFail (tstate=0x7fa01c8142a0 <_PyRuntime+315680>) at Python/gc.c:1703 #14 finalize_modules (tstate=0x7fa01c8142a0 <_PyRuntime+315680>) at Python/pylifecycle.c:1749 #15 0x00007fa01c4e109a in _Py_Finalize.constprop.0 (runtime=<optimized out>) at Python/pylifecycle.c:2144 #16 0x00007fa01c4a796a in Py_RunMain () at Modules/main.c:778 #17 0x00007fa01c4a167c in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:830 #18 0x00007fa01be27741 in __libc_start_call_main (main=main@entry=0x5619d7ec2120 <main>, argc=argc@entry=5, argv=argv@entry=0x7ffc1770a3d8) at ../sysdeps/nptl/libc_start_call_main.h:59 #19 0x00007fa01be27879 in __libc_start_main_impl (main=0x5619d7ec2120 <main>, argc=5, argv=0x7ffc1770a3d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc1770a3c8) at ../csu/libc-start.c:360 #20 0x00005619d7ec2045 in _start () (gdb) f 1 #1 0x00007fa01c3a49e8 in type_call (self=0x7fa01c7aa380 <PyDateTime_DeltaType.lto_priv.0>, args=0x7fa01c7dd390 <_PyRuntime+90640>, kwds=0x7fa0167a32c0) at Objects/typeobject.c:2372 } (gdb) p ((PyTypeObject *) self)->tp_name $1 = 0x7fa01c5a13a9 "datetime.timedelta" (gdb) p ((PyTupleObject *) args)->ob_base.ob_size $2 = 0 (gdb) p ((PyDictObject *) kwds)->ma_keys->dk_nentries $3 = 1 (gdb) p (char *) (((PyASCIIObject *) ((PyDictUnicodeEntry *) &((int8_t*) ((PyDictObject *) kwds)->ma_keys->dk_indices)[8])->me_key) + 1) $4 = 0x7fa01bd153a8 "seconds" (gdb) p ((PyFloatObject *) ((PyDictUnicodeEntry *) &((int8_t*) ((PyDictObject *) kwds)->ma_keys->dk_indices)[8])->me_value)->ob_fval $5 = 0@marmitar Can you provide a MRE?
Sorry about earlier, I forgot the backtrace (didn't know about
gdb/libpython.py). The issue is still withtqdm, specifically with this usage here:The minimal reproducer I got is this (with Python 3.14.5 and tqdm 4.67.3 & 4.68.0):
import tqdm bar = None def main(): global bar bar = tqdm.tqdm(total=100) bar.update(1) main()
If you need the same software stack I used, you can use the Arch Linux Archive in a Docker image:
Dockerfile
# syntax=docker/dockerfile:1 FROM docker.io/archlinux/archlinux:base-20260604.0.541199 RUN --mount=type=cache,sharing=locked,target=/var/cache/pacman <<EOF set -eux printf 'Server = https://archive.archlinux.org/repos/2026/06/04/$repo/os/$arch\n' > /etc/pacman.d/mirrorlist pacman -Syu --noconfirm python-tqdm EOF RUN cat > mre.py <<EOF import tqdm bar = None def main(): global bar bar = tqdm.tqdm(total=100) bar.update(1) main() EOF CMD ["python", "mre.py"]
- added a commit that references this issue
on Sep 3, 2026
Crash report
What happened?
I was testing some AI code with ollama and stumbled across a really weird crash.
The fact that it happens during an IndexError and a specific function has to be there leads me to believe that this is a CPython bug and not an ollama bug.
PythonCore.zip
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.13.2 (main, Feb 5 2025, 08:05:21) [GCC 14.2.1 20250128]
Linked PRs