Skip to content

CUDA: Add optional Driver API argument logging #6798

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sklam merged 1 commit into from
Mar 8, 2021
Merged

CUDA: Add optional Driver API argument logging #6798

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
CUDA: Add optional Driver API argument logging 
Setting the config variable `CUDA_LOG_API_ARGS` enables logging of
Driver API function arguments in the CUDA logs.

This commit also documents this variable along with the CUDA_LOG_LEVEL
config variable.

Without the API argument logging, logs look like (for example):

```
$ NUMBA_CUDA_LOG_LEVEL=DEBUG python -c "from numba import cuda; cuda.device_array(1)"
== CUDA [237]  INFO -- init
== CUDA [238] DEBUG -- call driver api: cuInit
== CUDA [275] DEBUG -- call driver api: cuCtxGetCurrent
== CUDA [275] DEBUG -- call driver api: cuCtxGetCurrent
== CUDA [275] DEBUG -- call driver api: cuDeviceGetCount
== CUDA [275] DEBUG -- call driver api: cuDeviceGet
== CUDA [275] DEBUG -- call driver api: cuDeviceComputeCapability
== CUDA [275] DEBUG -- call driver api: cuDeviceGetName
== CUDA [275] DEBUG -- call driver api: cuDeviceGetUuid
== CUDA [275] DEBUG -- call driver api: cuDeviceGet
== CUDA [275] DEBUG -- call driver api: cuDeviceComputeCapability
== CUDA [275] DEBUG -- call driver api: cuDeviceGetName
== CUDA [275] DEBUG -- call driver api: cuDeviceGetUuid
== CUDA [276] DEBUG -- call driver api: cuDevicePrimaryCtxRetain
== CUDA [370] DEBUG -- call driver api: cuCtxPushCurrent_v2
== CUDA [370] DEBUG -- call driver api: cuMemGetInfo_v2
== CUDA [371] DEBUG -- call driver api: cuMemAlloc_v2
== CUDA [371]  INFO -- add pending dealloc: cuMemFree_v2 8 bytes
```

With API argument logging, this becomes:

```
$ NUMBA_CUDA_LOG_API_ARGS=1 NUMBA_CUDA_LOG_LEVEL=DEBUG python -c "from numba import cuda; cuda.device_array(1)"
== CUDA [242]  INFO -- init
== CUDA [242] DEBUG -- call driver api: cuInit(0)
== CUDA [276] DEBUG -- call driver api: cuCtxGetCurrent(<cparam 'P' (0x7ff5954c9310)>)
== CUDA [276] DEBUG -- call driver api: cuCtxGetCurrent(<cparam 'P' (0x7ff5954c9310)>)
== CUDA [276] DEBUG -- call driver api: cuDeviceGetCount(<cparam 'P' (0x7ff5954c9310)>)
== CUDA [276] DEBUG -- call driver api: cuDeviceGet(<cparam 'P' (0x7ff5954c9310)>, 0)
== CUDA [276] DEBUG -- call driver api: cuDeviceComputeCapability(<cparam 'P' (0x7ff5953faa90)>, <cparam 'P' (0x7ff5953fa990)>, 0)
== CUDA [277] DEBUG -- call driver api: cuDeviceGetName(<numba.cuda.cudadrv.driver.c_char_Array_128 object at 0x7ff5953fabc0>, 128, 0)
== CUDA [277] DEBUG -- call driver api: cuDeviceGetUuid(<cparam 'P' (0x7ff5953fad90)>, 0)
== CUDA [277] DEBUG -- call driver api: cuDeviceGet(<cparam 'P' (0x7ff5954c9310)>, 1)
== CUDA [277] DEBUG -- call driver api: cuDeviceComputeCapability(<cparam 'P' (0x7ff5953fad10)>, <cparam 'P' (0x7ff5953fad90)>, 1)
== CUDA [277] DEBUG -- call driver api: cuDeviceGetName(<numba.cuda.cudadrv.driver.c_char_Array_128 object at 0x7ff5953fabc0>, 128, 1)
== CUDA [277] DEBUG -- call driver api: cuDeviceGetUuid(<cparam 'P' (0x7ff5953faa90)>, 1)
== CUDA [277] DEBUG -- call driver api: cuDevicePrimaryCtxRetain(<cparam 'P' (0x7ff5954c9310)>, 0)
== CUDA [367] DEBUG -- call driver api: cuCtxPushCurrent_v2(c_void_p(94307166233376))
== CUDA [367] DEBUG -- call driver api: cuMemGetInfo_v2(<cparam 'P' (0x7ff5953fa990)>, <cparam 'P' (0x7ff5953faa90)>)
== CUDA [367] DEBUG -- call driver api: cuMemAlloc_v2(<cparam 'P' (0x7ff5953d3310)>, 8)
== CUDA [367]  INFO -- add pending dealloc: cuMemFree_v2 8 bytes
```
  • Loading branch information
@gmarkall
gmarkall committed Mar 5, 2021
commit f01d12adb4d55efc4da679825d606e18e71e7e8b
14 changes: 14 additions & 0 deletions docs/source/reference/envvars.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,20 @@ GPU support
takes place, and the user of Numba (and other CUDA libraries) is responsible
for ensuring correctness with respect to synchronization on streams.

.. envvar:: NUMBA_CUDA_LOG_LEVEL

For debugging purposes. If no other logging is configured, the value of this
variable is the logging level for CUDA API calls. The default value is
``CRITICAL`` - to trace all API calls on standard error, set this to
``DEBUG``.

.. envvar:: NUMBA_CUDA_LOG_API_ARGS

By default the CUDA API call logs only give the names of functions called.
Setting this variable to 1 also includes the values of arguments to Driver
API calls in the logs.


Threading Control
-----------------

Expand Down
3 changes: 3 additions & 0 deletions numba/core/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,9 @@ def avx_default():
# Any existing logging configuration is preserved.
CUDA_LOG_LEVEL = _readenv("NUMBA_CUDA_LOG_LEVEL", str, '')

# Include argument values in the CUDA Driver API logs
CUDA_LOG_API_ARGS = _readenv("NUMBA_CUDA_LOG_API_ARGS", int, 0)

# Maximum number of pending CUDA deallocations (default: 10)
CUDA_DEALLOCS_COUNT = _readenv("NUMBA_CUDA_MAX_PENDING_DEALLOCS_COUNT",
int, 10)
Expand Down
15 changes: 13 additions & 2 deletions numba/cuda/cudadrv/driver.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,12 +293,23 @@ def __getattr__(self, fname):
return safe_call

def _wrap_api_call(self, fname, libfn):
@functools.wraps(libfn)
def verbose_cuda_api_call(*args):
argstr = ", ".join([str(arg) for arg in args])
_logger.debug('call driver api: %s(%s)', libfn.__name__, argstr)
retcode = libfn(*args)
self._check_error(fname, retcode)

def safe_cuda_api_call(*args):
_logger.debug('call driver api: %s', libfn.__name__)
retcode = libfn(*args)
self._check_error(fname, retcode)
return safe_cuda_api_call

if config.CUDA_LOG_API_ARGS:
wrapper = verbose_cuda_api_call
else:
wrapper = safe_cuda_api_call

return functools.wraps(libfn)(wrapper)

def _find_api(self, fname):
# Try version 2
Expand Down