LLVM backend code generation error on ARM64

Using the latest 0.63.1 build, running the following code:

from numba import guvectorize, int64
import numpy as np

@guvectorize([(int64[:], int64, int64[:])], '(n),()->(n)')
def g(x, y, res):
    for i in range(x.shape[0]):
        res[i] = x[i] + y

a = np.arange(5)
result = g(a, 2) #type: ignore
print(result)

Results in the following error:

DefIdx 1 exceeds machine model writes for renamable $x11, renamable $x10 = LDPXi $sp, 0 :: (load (s64) from %fixed-stack.7), (load (s64) from %fixed-stack.8, align 16)
 (Try with MCSchedModel.CompleteModel set to false)incomplete machine model
UNREACHABLE executed at /root/miniconda3/envs/buildenv/conda-bld/llvmdev_1755759150100/work/llvm/lib/CodeGen/TargetSchedule.cpp:229!
Aborted (core dumped)

I can get Numba to work by including this code snippet above the import statements:

import os
# ARM64 LLVM optimization workaround
# Use generic ARM64 target to avoid CPU-specific scheduling model bugs
os.environ['NUMBA_CPU_NAME'] = 'generic'
from numba import guvectorize, int64
import numpy as np

Since this issue seems to be specific to my CPU model, I've included the details from cat /proc/cpuinfo | grep "CPU":

processor       : 0
BogoMIPS        : 38.40
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 asimdfhm uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 frint i8mm bf16 rng afp rpres
CPU implementer : 0x51
CPU architecture: 8
CPU variant     : 0x2
CPU part        : 0x001
CPU revision    : 1

I have a 12 core CPU, I've included the details for the first core for brevity.

I'm willing to provide additional details if necessary.

Thank you for submitting this error report. Can you run numba -s and post the output. Also, what OS/Platform is this? Linux?

@esc

@esc Yes, this is on Linux. Here is my numba -s output.

System info:
--------------------------------------------------------------------------------
__Time Stamp__
Report started (local time)                   : 2025-12-20 10:12:25.333612
UTC start time                                : 2025-12-20 15:12:25.333676
Running time (s)                              : 0.789041

__Hardware Information__
Machine                                       : aarch64
CPU Name                                      : oryon-1
CPU Count                                     : 12
Number of accessible CPUs                     : 12
List of accessible CPUs cores                 : 0 1 2 3 4 5 6 7 8 9 10 11
CFS Restrictions (CPUs worth of runtime)      : None

CPU Features                                  : aes crc fp-armv8 lse neon sha2

Memory Total (MB)                             : 15744
Memory Available (MB)                         : 12676

__OS Information__
Platform Name                                 : Linux-6.6.87.2-microsoft-standard-WSL2-aarch64-with-glibc2.39
Platform Release                              : 6.6.87.2-microsoft-standard-WSL2
OS Name                                       : Linux
OS Version                                    : #1 SMP PREEMPT_DYNAMIC Thu Jun  5 18:31:42 UTC 2025
OS Specific Version                           : ?
Libc Version                                  : glibc 2.39

__Python Information__
Python Compiler                               : GCC 13.3.0
Python Implementation                         : CPython
Python Version                                : 3.12.3
Python Locale                                 : C.UTF-8

__Numba Toolchain Versions__
Numba Version                                 : 0.63.1
llvmlite Version                              : 0.46.0

__LLVM Information__
LLVM Version                                  : 20.1.8

__CUDA Information__
CUDA Target Implementation                    : Built-in
CUDA Device Initialized                       : False
CUDA Driver Version                           : ?
CUDA Runtime Version                          : ?
CUDA NVIDIA Bindings Available                : ?
CUDA NVIDIA Bindings In Use                   : ?
CUDA Minor Version Compatibility Available    : ?
CUDA Minor Version Compatibility Needed       : ?
CUDA Minor Version Compatibility In Use       : ?
CUDA Detect Output:
None
CUDA Libraries Test Output:
None

__NumPy Information__
NumPy Version                                 : 2.3.4
NumPy Supported SIMD features                 : ('NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD', 'FPHP', 'ASIMDHP', 'ASIMDDP', 'ASIMDFHM')
NumPy Supported SIMD dispatch                 : ('ASIMDHP', 'ASIMDFHM', 'SVE')
NumPy Supported SIMD baseline                 : ('NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD')
NumPy AVX512_SKX support detected             : False

__SVML Information__
SVML State, config.USING_SVML                 : False
SVML Library Loaded                           : False
llvmlite Using SVML Patched LLVM              : False
SVML Operational                              : False

__Threading Layer Information__
TBB Threading Layer Available                 : False
+--> Disabled due to Unknown import problem.
OpenMP Threading Layer Available              : False
+--> Disabled due to Unknown import problem.
Workqueue Threading Layer Available           : True
+-->Workqueue imported successfully.

__Numba Environment Variable Information__
None found.

__Conda Information__
Conda not available.

__Installed Packages__
Package  Version
-------- -------
llvmlite 0.46.0
numba    0.63.1
numpy    2.3.4
pip      24.0
psutil   7.1.3

No errors reported.


__Warning log__
Warning (cuda): CUDA driver library cannot be found or no CUDA enabled devices are present.
Exception class: <class 'numba.cuda.cudadrv.error.CudaSupportError'>
Warning: Conda not available.
 Error was [Errno 2] No such file or directory: 'conda'

Warning (no file): /sys/fs/cgroup/cpuacct/cpu.cfs_quota_us
Warning (no file): /sys/fs/cgroup/cpuacct/cpu.cfs_period_us
--------------------------------------------------------------------------------
If requested, please copy and paste the information between
the dashed (----) lines, or from a given specific section as
appropriate.

=============================================================
IMPORTANT: Please ensure that you are happy with sharing the
contents of the information present, any information that you
wish to keep private you should remove before sharing.
=============================================================

@Detrilemma

@Detrilemma OK, thank you. Nothing obvious stands out. The project is currently on winter break so you are likely to experience some delay with the triage. I expect that this will not be triaged until mid-january 2026 at the earliest. Thank you in advance for your patience.

@Detrilemma

@Detrilemma took another look today. This is not "just" linux, this is WSL, i.e. Windows Subsytem on Linux? Also, it looks like you compiled the stack yourself, given the llvm is in the conda-bld directory? Did you compile LLVM yourself too?

Looks like you hit the following snippet:

Yes, I'm running Linux using WSL. I didn't compile anything myself in this environment. I created a separate venv, pip installed numba, and the copied some code from the docs to try and isolate this issue. If you think it would be helpful, I could try and re-create this issue in a pure Windows environment to see if the issue is specific to my particular CPU model.

I am not sure why the following path shows up in the error message. I think this might be artifact of compiling llvmlite, not sure:

/root/miniconda3/envs/buildenv/conda-bld/llvmdev_1755759150100/work/llvm/lib/CodeGen/TargetSchedule.cpp:229!

We recently saw some micro-architecture related issues.

#10325

So, yes, if you can try on plain windows that would help and add additional data to this.

I started setting up a Windows environment to test this only to find that Numba doesn't support Windows ARM64. This makes sense given that LLVMlite also does not support Windows ARM64.

As a sanity check, I went ahead and installed Numba to an environment running the amd64 Python interpreter. This worked, however this come to no surprise and only confirms that the x86 to ARM64 emulator is functioning correctly.

I started setting up a Windows environment to test this only to find that Numba doesn't support Windows ARM64. This makes sense given that LLVMlite also does not support Windows ARM64.

As a sanity check, I went ahead and installed Numba to an environment running the amd64 Python interpreter. This worked, however this come to no surprise and only confirms that the x86 to ARM64 emulator is functioning correctly.

Yes, Windows-on-arm is currently WIP. You can track the issue here: #10066 -- hopefully support will become available sometime in 2026.

I can reproduce this on a Jetson AGX Orin (a 12-core NVIDIA ARM CPU) by setting the CPU name to match that given in the report:

NUMBA_CPU_NAME="oryon-1" python repro.py 
DefIdx 1 exceeds machine model writes for renamable $x11, renamable $x10 = LDPXi $sp, 0 :: (load (s64) from %fixed-stack.7), (load (s64) from %fixed-stack.8, align 16)
 (Try with MCSchedModel.CompleteModel set to false)incomplete machine model
UNREACHABLE executed at /root/miniconda3/envs/buildenv/conda-bld/llvmdev_1755759150100/work/llvm/lib/CodeGen/TargetSchedule.cpp:229!

This may be because we build with assertions enabled. It appears that without assertions, the function reporting this error will return a default latency if one does does not exist in the model.

With an LLVM build with assertions enabled I can reproduce the issue with llc (using the IR dumped with NUMBA_DUMP_LLVM=1:

/opt/llvm/20.1.8/bin/llc -mcpu=oryon-1 -O3 repro.ll 
DefIdx 1 exceeds machine model writes for $q1, renamable $q0 = LDPQi $sp, 2 :: (load (s128) from %stack.20), (load (s128) from %stack.19)
 (Try with MCSchedModel.CompleteModel set to false)incomplete machine model
UNREACHABLE executed at /home/gmarkall/numbadev/llvm-project/llvm/lib/CodeGen/TargetSchedule.cpp:229!
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: /opt/llvm/20.1.8/bin/llc -mcpu=oryon-1 -O3 repro.ll
1.	Running pass 'Function Pass Manager' on module 'repro.ll'.
2.	Running pass 'PostRA Machine Instruction Scheduler' on function '@_ZN8__main__1gB2v1B52c8tJTIeFImbKsAWlBR0FJMfbgsMFGDwYEQhVCwwAMEsTAA_3d_3dE5ArrayIxLi1E1A7mutable7alignedEx5ArrayIxLi1E1A7mutable7alignedE'
 #0 0x0000aaaac32a4458 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /home/gmarkall/numbadev/llvm-project/llvm/lib/Support/Unix/Signals.inc:799:22
 #1 0x0000aaaac32a4810 PrintStackTraceSignalHandler(void*) /home/gmarkall/numbadev/llvm-project/llvm/lib/Support/Unix/Signals.inc:875:1
 #2 0x0000aaaac32a1f74 llvm::sys::RunSignalHandlers() /home/gmarkall/numbadev/llvm-project/llvm/lib/Support/Signals.cpp:105:20
 #3 0x0000aaaac32a3c50 SignalHandler(int) /home/gmarkall/numbadev/llvm-project/llvm/lib/Support/Unix/Signals.inc:415:1
 #4 0x0000ffff9c8b27bc (linux-vdso.so.1+0x7bc)
 #5 0x0000ffff9c3ef200 __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #6 0x0000ffff9c3aa67c gsignal ./signal/../sysdeps/posix/raise.c:27:6
 #7 0x0000ffff9c397130 abort ./stdlib/abort.c:81:7
 #8 0x0000aaaac31c7af8 bindingsErrorHandler(void*, char const*, bool) /home/gmarkall/numbadev/llvm-project/llvm/lib/Support/ErrorHandling.cpp:223:55
 #9 0x0000aaaac1f7ef48 llvm::TargetSchedModel::computeOperandLatency(llvm::MachineInstr const*, unsigned int, llvm::MachineInstr const*, unsigned int) const /home/gmarkall/numbadev/llvm-project/llvm/lib/CodeGen/TargetSchedule.cpp:235:28
#10 0x0000aaaac1e64b50 llvm::ScheduleDAGInstrs::addPhysRegDataDeps(llvm::SUnit*, unsigned int) /home/gmarkall/numbadev/llvm-project/llvm/lib/CodeGen/ScheduleDAGInstrs.cpp:281:23
#11 0x0000aaaac1e65048 llvm::ScheduleDAGInstrs::addPhysRegDeps(llvm::SUnit*, unsigned int) /home/gmarkall/numbadev/llvm-project/llvm/lib/CodeGen/ScheduleDAGInstrs.cpp:347:27
#12 0x0000aaaac1e66d34 llvm::ScheduleDAGInstrs::buildSchedGraph(llvm::AAResults*, llvm::RegPressureTracker*, llvm::PressureDiffs*, llvm::LiveIntervals*, bool) /home/gmarkall/numbadev/llvm-project/llvm/lib/CodeGen/ScheduleDAGInstrs.cpp:859:23
#13 0x0000aaaac1c1858c llvm::ScheduleDAGMI::schedule() /home/gmarkall/numbadev/llvm-project/llvm/lib/CodeGen/MachineScheduler.cpp:837:17
#14 0x0000aaaac1c17a40 (anonymous namespace)::MachineSchedulerBase::scheduleRegions(llvm::ScheduleDAGInstrs&, bool) /home/gmarkall/numbadev/llvm-project/llvm/lib/CodeGen/MachineScheduler.cpp:653:27
#15 0x0000aaaac1c17128 (anonymous namespace)::PostMachineScheduler::runOnMachineFunction(llvm::MachineFunction&) /home/gmarkall/numbadev/llvm-project/llvm/lib/CodeGen/MachineScheduler.cpp:498:23
#16 0x0000aaaac1ae47cc llvm::MachineFunctionPass::runOnFunction(llvm::Function&) /home/gmarkall/numbadev/llvm-project/llvm/lib/CodeGen/MachineFunctionPass.cpp:94:33
#17 0x0000aaaac2444bb8 llvm::FPPassManager::runOnFunction(llvm::Function&) /home/gmarkall/numbadev/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1406:40
#18 0x0000aaaac2444e00 llvm::FPPassManager::runOnModule(llvm::Module&) /home/gmarkall/numbadev/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1452:29
#19 0x0000aaaac24451ec (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) /home/gmarkall/numbadev/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1521:38
#20 0x0000aaaac2440c44 llvm::legacy::PassManagerImpl::run(llvm::Module&) /home/gmarkall/numbadev/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:539:55
#21 0x0000aaaac24459e8 llvm::legacy::PassManager::run(llvm::Module&) /home/gmarkall/numbadev/llvm-project/llvm/lib/IR/LegacyPassManager.cpp:1648:17
#22 0x0000aaaac09b79f8 compileModule(char**, llvm::LLVMContext&) /home/gmarkall/numbadev/llvm-project/llvm/tools/llc/llc.cpp:753:34
#23 0x0000aaaac09b5bc4 main /home/gmarkall/numbadev/llvm-project/llvm/tools/llc/llc.cpp:411:35
#24 0x0000ffff9c3973fc __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:74:3
#25 0x0000ffff9c3974cc call_init ./csu/../csu/libc-start.c:128:20
#26 0x0000ffff9c3974cc __libc_start_main ./csu/../csu/libc-start.c:379:5
#27 0x0000aaaac09b4a30 _start (/opt/llvm/20.1.8/bin/llc+0x7e4a30)
Aborted (core dumped)

A similar issue occurs on LLVM main.

The following crude change works around the issue:

diff --git a/llvm/lib/Target/AArch64/AArch64SchedOryon.td b/llvm/lib/Target/AArch64/AArch64SchedOryon.td
index 435eaf99c617..9fb685283829 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedOryon.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedOryon.td
@@ -21,7 +21,7 @@ def OryonModel : SchedMachineModel {
   let MispredictPenalty     =  13; // 13 cycles for mispredicted branch.
   let LoopMicroOpBufferSize =   16; // Oryon-1 does not have loop micro op buffer, we enable this pseudo value to allow for aggressive unrolling based on runtime TC.
   let PostRAScheduler       =   1; // Using PostRA sched.
-  let CompleteModel         =   1;
+  let CompleteModel         =   0;
 
   list<Predicate> UnsupportedFeatures = !listconcat(SVEUnsupported.F,
                                                     SMEUnsupported.F,

Perhaps a slightly better patch, though I am wading out of my depth here:

diff --git a/llvm/lib/Target/AArch64/AArch64SchedOryon.td b/llvm/lib/Target/AArch64/AArch64SchedOryon.td
index 435eaf99c617..0886559ef1fa 100644
--- a/llvm/lib/Target/AArch64/AArch64SchedOryon.td
+++ b/llvm/lib/Target/AArch64/AArch64SchedOryon.td
@@ -443,64 +443,79 @@ def ORYONWrite_5Cyc_10Uops_LD_I012345 : SchedWriteRes<[ORYONLD, ORYONI012345]> {
 def ORYONWrite_1Cyc_ST : SchedWriteRes<[ORYONST]>;
 
 def ORYONWrite_1Cyc_2Uops_ST : SchedWriteRes<[ORYONST]> {
+  let Latency = 5;
   let NumMicroOps = 2;
 }
 
 def ORYONWrite_1Cyc_3Uops_ST : SchedWriteRes<[ORYONST]> {
+  let Latency = 5;
   let NumMicroOps = 3;
 }
 
 def ORYONWrite_1Cyc_4Uops_ST : SchedWriteRes<[ORYONST]> {
+  let Latency = 5;
   let NumMicroOps = 4;
 }
 
 def ORYONWrite_1Cyc_5Uops_ST : SchedWriteRes<[ORYONST]> {
+  let Latency = 5;
   let NumMicroOps = 5;
 }
 
 def ORYONWrite_1Cyc_6Uops_ST : SchedWriteRes<[ORYONST]> {
+  let Latency = 5;
   let NumMicroOps = 6;
 }
 
 def ORYONWrite_1Cyc_8Uops_ST : SchedWriteRes<[ORYONST]> {
+  let Latency = 5;
   let NumMicroOps = 8;
 }
 
 def ORYONWrite_1Cyc_10Uops_ST : SchedWriteRes<[ORYONST]> {
+  let Latency = 5;
   let NumMicroOps = 10;
 }
 
 // 1 cycle for neon write: float + ASIMD with Post/Pre Inc/Dec access
 // also includes Pair store until further informed
 def ORYONWrite_1Cyc_ST_I012345 : SchedWriteRes<[ORYONST, ORYONI012345]> {
+  let Latency = 5;
   let NumMicroOps = 3;
 }
 
 def ORYONWrite_1Cyc_2Uops_ST_I012345 : SchedWriteRes<[ORYONST, ORYONI012345]> {
+  let Latency = 5;
   let NumMicroOps = 2;
 }
 
 def ORYONWrite_1Cyc_3Uops_ST_I012345 : SchedWriteRes<[ORYONST, ORYONI012345]> {
+  let Latency = 5;
   let NumMicroOps = 3;
 }
 
 def ORYONWrite_1Cyc_4Uops_ST_I012345 : SchedWriteRes<[ORYONST, ORYONI012345]> {
+  let Latency = 5;
   let NumMicroOps = 4;
 }
 
 def ORYONWrite_1Cyc_5Uops_ST_I012345 : SchedWriteRes<[ORYONST, ORYONI012345]> {
+  let Latency = 5;
   let NumMicroOps = 5;
 }
 
 def ORYONWrite_1Cyc_6Uops_ST_I012345 : SchedWriteRes<[ORYONST, ORYONI012345]> {
+  let Latency = 5;
   let NumMicroOps = 6;
 }
 
 def ORYONWrite_1Cyc_8Uops_ST_I012345 : SchedWriteRes<[ORYONST, ORYONI012345]> {
+  let Latency = 5;
   let NumMicroOps = 8;
 }
 
 def ORYONWrite_1Cyc_10Uops_ST_I012345 : SchedWriteRes<[ORYONST, ORYONI012345]> {
+  let Latency = 5;
   let NumMicroOps = 10;
 }

... those latencies should all be 1, not 5 🤕

... those latencies should all be 1, not 5 🤕

It turns out that reducing the latencies to 1 causes the issue to reproduce again. So maybe they shouldn't be 1 just to match the 1Cyc description in their name. (Perhaps the default latency is 1 anyway?)

From triage meeting: Only we build LLVM with assertions enabled, so applying #10388 (comment) to our llvmdev build is likely an acceptable workaround.

@gmarkall

@gmarkall that's good to hear. Are there any plans to change the llvmdev build so that it's configured for this particular case upon installation? I don't expect your team to prioritize my odd case; I’m just curious if there is a long-term plan.

We intend to change the llvmdev build, but it can't be done at installation time - we need to do a new build of the llvmdev package with the patch applied. For that, we need to do a PR that modifies the llvmdev recipe to apply the patch, and then to do a new build of it.

patch to flip CompleteModel field to resolve this issue has been merged to llvmdev recipe - numba/llvmlite#1472. I'm working to rebuild llvmdev and upload to numba channel now. This should resolve the issue on oryon-1 with next llvmlite and Numba release.

Navigation Menu

Repository navigation

LLVM backend code generation error on ARM64 #10388

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

LLVM backend code generation error on ARM64 #10388

Description

Activity

esc commented on Dec 20, 2025

Detrilemma commented on Dec 20, 2025

esc commented on Dec 20, 2025

esc commented on Dec 21, 2025

Detrilemma commented on Dec 21, 2025

esc commented on Dec 21, 2025

Detrilemma commented on Dec 24, 2025

esc commented on Dec 30, 2025

gmarkall commented on Jan 5, 2026

gmarkall commented on Jan 5, 2026

gmarkall commented on Jan 5, 2026

gmarkall commented on Jan 5, 2026

gmarkall commented on Jan 5, 2026

gmarkall commented on Jan 5, 2026

gmarkall commented on Jan 5, 2026

gmarkall commented on Jan 5, 2026

gmarkall commented on Jan 20, 2026

Detrilemma commented on Jan 21, 2026

gmarkall commented on Jan 21, 2026

swap357 commented on Aug 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions