Support non-constant exception values in JIT - #8134

What are the situations in which someone can reraise an exception? I see that code like this below won't work because we can't do as e

import numba

@numba.njit
def f(lst):
    try:
        idx = lst[10]
    except KeyError as e:
        raise e

f([1])

This sort of thing is permissible:

from numba import njit

@njit
def reraise():
    raise

def foo():
    try:
        raise ZeroDivisionError
    except ZeroDivisionError:
        reraise()

foo()

This is a good question. Adding this comment for visibility in future reviews.

What free's this allocation?

I've changed the code to use the NRT allocator, which I believe should manage the memory. Correct?

The NRT allocator is this:

So, the memory is still leaking?

Hi @stuartarchibald, once you have the chance, can you review this PR? I believe I have addressed the memory leak issue

The following code makes a lot of use of the pyapi without checking if the function calls were successful?

Fixed

Examples of things that are currently not working correctly:

Unhashable:

from numba import njit
import numpy as np

@njit
def foo(z):
    raise ValueError('a', [1,])

foo(1)

Unhashable:

from numba import njit
import numpy as np

@njit
def foo(z):
    raise ValueError('a', ValueError('a'))

foo(1)

Incorrect arg reported: ValueError: i8* null

from numba import njit
import numpy as np

@njit
def foo(pred):
    raise ValueError({'a': 1, 'b': np.ones(4)})

foo(1)

Incorrect arg reported: ValueError: i8* null

from numba import njit
import numpy as np

@njit
def foo():
    raise ValueError(range(3))

foo()

Segfaults

from numba import njit
import numpy as np

@njit
def foo():
    raise ValueError({'a': 1, 'b': 2})

foo()

Segfaults:

from numba import njit
import numpy as np

@njit
def foo(rng):
    raise ValueError(rng.bit_generator)

rng = np.random.default_rng()
foo(rng)

Unhashable:

from numba import njit
import numpy as np

thing = np.ones(10)

@njit
def foo():
    raise ValueError(thing)

foo()

Unhashable:

from numba import njit, objmode
import numpy as np

@njit
def foo():
    raise ValueError(slice(1, 2))


foo()

The cases you marked as unhashable are not related to this PR as Numba type them as static raise. As for the other four cases, the errors are correctly treated.

Why does DynamicTryRaise inherit from StaticTryRaise? Should DynamicTryRaise and StaticTryRaise both inherit from a _TryRaise ABC instead?

DynamicTryRaise has the same methods and values as StaticTryRaise. If needed, I can just copy instead of inheriting.

I think copy or create a common base, either is fine. Was just trying to avoid what I think would seem to be an unexpected inheritance relationship.

I'll just copy. It's better than creating a forth *Raise class in ir.py.

This fails if the argument to try_raise is a np.array (I tried a = np.ones(3)). There is an AttributeError in numba.core.boxing.box_array, line 429:

AttributeError: 'NoneType' object has no attribute 'read_cost'

(suspect this is because the env_manager is None?)

The same thing also happens with the more simple:

@njit
def raise_(x):
    raise ValueError(x)

@njit
def foo(x):
    raise_(x)

foo(np.ones(3))

I've got to what I think is the cause of this. Essentially when the code for unwrapping the dynamic exception is generated it has to emit code to box native representations into python objects, but the "environment manager" is missing as kwarg "env_manager=" here:

You can get an environment manager by calling self.context.get_env_manager(builder) but this isn't enough though because the emitted environment sentry needs to be aware that this is a Python context code region. I made this change locally to be able to wire this through:

@@ -123,13 +123,14 @@ class CPUContext(BaseContext):
             builder, envptr, _dynfunc._impl_info['offsetof_env_body'])
         return EnvBody(self, builder, ref=body_ptr, cast_ref=True)
 
-    def get_env_manager(self, builder):
+    def get_env_manager(self, builder, return_pyobject=False):
         envgv = self.declare_env_global(builder.module,
                                         self.get_env_name(self.fndesc))
         envarg = builder.load(envgv)
         pyapi = self.get_python_api(builder)
         pyapi.emit_environment_sentry(
             envarg, debug_msg=self.fndesc.env_name,
+            return_pyobject=return_pyobject
         )
         env_body = self.get_env_body(builder, envarg)
         return pyapi.get_env_manager(self.environment, env_body, envarg)

and then did this:

@@ -576,10 +576,12 @@ class CPUCallConv(BaseCallConv):
         nb_types = [typ for typ in nb_types if typ is not None]
 
         # convert native values into CPython objects
+        env_manager = self.context.get_env_manager(builder,
+                                                   return_pyobject=True)
         objs = []
         for i, typ in enumerate(nb_types):
             val = builder.extract_value(builder.load(st_ptr), i)
-            obj = pyapi.from_native_value(typ, val)
+            obj = pyapi.from_native_value(typ, val, env_manager=env_manager)

thoughts?

oh, that's fine

Thanks @stuartarchibald

Thanks @stuartarchibald

No problem.

I'm not sure that this has any effect unless the error state is communicated through a return value of NULL.