Add debug infomation support based on datamodel. by stuartarchibald · Pull Request #7177 · numba/numba

stuartarchibald · 2021-07-02T17:22:24Z

This:

Adds debug info for types that are based on DataModel by emitting information to make them appear as structures.
Fixes another bug in lineinfo emission where decref calls at the end of basic blocks were having line information associated with them from the variable declaration site.

Impact of 1. (and 2.!, to sort the lineinfo out) is demonstrated in the following:

$ cat debuginfo_structures.py
from numba import njit
import numpy as np

@njit(debug=True)
def foo():
    x = np.arange(21, dtype=np.float32).reshape((7, 3))
    y = (1, 2, 'x', 4)
    z = "some string" + str(10) + "some other string"
    return x, y, z

foo()

then running the above under gdb:

$ NUMBA_OPT=0 gdb -q  -ex 'b "_ZN8__main__7foo$241E"' -ex 'r' --args python debuginfo_structures.py 
<snip>

Breakpoint 1, __main__::foo$241 () at debuginfo_structures.py:4
4       @njit(debug=True)

(gdb) n
6           x = np.arange(21, dtype=np.float32).reshape((7, 3))
(gdb) n
7           y = (1, 2, 'x', 4)
(gdb) n
8           z = "some string" + str(10) + "some other string"
(gdb) n
9           return x, y, z

(gdb) info locals
x = {meminfo = 0x555556c4a150 "\001", parent = 0x0, nitems = 21, itemsize = 4, data = 0x555556c4a180, shape = {7, 3}, 
  strides = {12, 4}}
y = {f0 = 1, f1 = 2, f2 = 0x0, f3 = 4}
z = {data = 0x555556c4a540 "some string10some other string", length = 30, kind = 1, is_ascii = 1, hash = 18446744073709551615, 
  meminfo = 0x555556c4a510 "\001", parent = 0x0}

(gdb) whatis x
type = array(float32, 2d, C) ({i8*, i8*, i64, i64, float*, [2 x i64], [2 x i64]})

(gdb) whatis y
type = Tuple(Literal[int](1), Literal[int](2), Literal[str](x), Literal[int](4)) ({i64, i64, i8*, i64})

(gdb) whatis z
type = unicode_type ({i8*, i64, i32, i32, i64, i8*, i8*})

(gdb) x.nitems
$1 = 21

(gdb) x/21f x.data # we know there's 21 items in x and x is float32 from whatis above
0x555556c4a180: 0       1       2       3
0x555556c4a190: 4       5       6       7
0x555556c4a1a0: 8       9       10      11
0x555556c4a1b0: 12      13      14      15
0x555556c4a1c0: 16      17      18      19
0x555556c4a1d0: 20
(gdb) c

Continuing.
[Inferior 1 (process 4124) exited normally]
(gdb) q

Note that:

Calling whatis on the variables will yield their Numba typing domain type repr and also their datamodel layout in LLVM IR.
That the structures are behaving like C structs and the members can be inspected.

As title.

This: * Adds a new env var ``NUMBA_EXTEND_VARIABLE_LIFETIMES`` which when set extends the lifetime of variables to the end of the block in which their lifetime ends. * Fixes up the Dels and their location information in the case of the above being set. * Wires the above into the post processor call in the legalizer. The legalizer is the only place which should change the variable life times as it is the last thing that is executed in the pipeline before lowering. Changing the lifetimes in general will break optimisation passes that rely on lifetime analysis via Del nodes. Fixes numba#7420

As title.

As title

As title.

This adds testing of the DWARF emission with gdb. It uses the gdb machine interface wrapped in pexpect to assert interaction behaviours. Some minor refactoring of the original gdb support is made to keep the gdb helper functions all in one place (the new gdb_support.py).

As title

stuartarchibald · 2021-09-29T08:43:31Z

8a7a77b failed on windows 64 py 37 np 117, doesn't seem related but capturing the output:

Starting: Test slice of test files
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.182.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
========================== Starting Command Output ===========================
"C:\Windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "D:\a\_temp\d7d104a6-8d26-4603-820b-57bcb9ecb958.cmd""
...................................................................s..........s.............s....s...................................................s..................................................sss..ss...............s.......................................................................s.......................................................................................................................................................................................sssssss.ssss........F.FF
skipped CUDA tests
skipped CUDA tests
Parallel: 469. Serial: 33
======================================================================
FAIL: test_float32_numpy2_fastmath_usecase (numba.tests.test_svml.TestSVMLGeneration)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\1\s\numba\tests\test_svml.py", line 245, in test_runner
    self.assertEqual(p.exitcode, 0, msg="process ended unexpectedly")
AssertionError: None != 0 : process ended unexpectedly

======================================================================
FAIL: test_float64_range4_fastmath_usecase (numba.tests.test_svml.TestSVMLGeneration)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\1\s\numba\tests\test_svml.py", line 245, in test_runner
    self.assertEqual(p.exitcode, 0, msg="process ended unexpectedly")
AssertionError: None != 0 : process ended unexpectedly

======================================================================
FAIL: test_int32_prange4_usecase (numba.tests.test_svml.TestSVMLGeneration)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\1\s\numba\tests\test_svml.py", line 245, in test_runner
    self.assertEqual(p.exitcode, 0, msg="process ended unexpectedly")
AssertionError: None != 0 : process ended unexpectedly

----------------------------------------------------------------------
Ran 502 tests in 840.808s

FAILED (failures=3, skipped=23)
##[error]Cmd.exe exited with code '1'.
Finishing: Test slice of test files

stuartarchibald · 2021-10-22T08:51:46Z

@sklam have merged in mainline so as to test with the changes to name mangling.

sklam · 2021-10-25T17:14:20Z

https://github.com/sklam/numba/tree/wip/dwarf_like_c fixes the GDB breakpoint stopping at function def line (def foo():) instead of the first line of the function body.

stuartarchibald · 2021-10-26T09:56:21Z

https://github.com/sklam/numba/tree/wip/dwarf_like_c fixes the GDB breakpoint stopping at function def line (def foo():) instead of the first line of the function body.

Thanks @sklam, well found! I've a couple of other fixes along with this to make the code gen match more what would be generated from C too. Think I'll combine with those and hopefully this issue will be resolved.

As title.

As title

as title

As title.

As title

sklam

Everything looks good. The improved GDB support will be a great help for users and developers to understand the emitted code.

sklam · 2021-10-26T14:44:26Z

I think we just need to verify that we didn't break cuda-gdb

sklam

cuda NvvmDIBuilder has became out-of-sync with the usage

gmarkall · 2021-10-26T19:58:33Z

cuda NvvmDIBuilder has became out-of-sync with the usage

I think we should get rid of it - NVVM uses the normal DIBuilder for NVVM 7. I seem to recall we don't support debug with NVVM 3.4 anymore, but will check if that was something in an unmerged PR.

sklam · 2021-10-26T23:44:18Z

CC @stuartarchibald @gmarkall

d8ef5b1 fixes the debuginfo emission for the CUDA target.

sklam

From OOB discussion, we will address the CUDA issues separately in #7514.

sklam · 2021-10-27T15:38:30Z

Checking buildfarm: numba_smoketest_cpu_yaml_58

sklam · 2021-10-27T19:45:08Z

Buildfarm passed. Manually confirmed execution of GDB test.

stuartarchibald added the 2 - In Progress label Jul 2, 2021

This was referenced Jul 2, 2021

Double function call in Numba IntelPython/numba-dpex#382

Closed

DEBUG: improve displaying complex data types IntelPython/numba-dpex#329

Closed

stuartarchibald added the Effort - long Long size effort needed label Jul 7, 2021

stuartarchibald mentioned this pull request Jul 9, 2021

Fixes for lineinfo emission #7196

Merged

stuartarchibald added 2 commits September 15, 2021 12:45

Add debug info for structured types based on datamodel.

06bb3a1

As title.

Add support for most simple data structures.

1e3e9d4

As title.

stuartarchibald force-pushed the wip/dwarf_datamodel_1 branch from 2d63784 to 1e3e9d4 Compare September 15, 2021 21:25

stuartarchibald added this to the Numba 0.55 RC milestone Sep 15, 2021

stuartarchibald added 3 commits September 16, 2021 13:06

Tests for basic types

1a97629

Add support for info args and fix up locals

41a7db0

As title.

This was referenced Sep 22, 2021

Show function arguments does not work for debugging w/ GDB #7414

Closed

Function argument are not available at the first line when debugging w/ GDB #7415

Closed

stuartarchibald added 7 commits September 22, 2021 15:19

Dedent the source string

4910c08

As title

Update test with new line info for function def

b299792

flake8

9cee639

Handle Python 3.7

7aa4d7f

Array type test.

06019be

As title.

Add dependencies for gdb testing

754fc98

As title

stuartarchibald added 7 commits September 29, 2021 09:54

Add test driver for gdb

ba84856

fix ptrace scope detection

e2a7518

Add more info to exceptions

ede7c8d

Increase timeout

92877b2

Add debug to output in gdb tests

2b428f1

Only run gdb on x86_64

9631fdf

Minor clean up of test

07d8992

stuartarchibald removed the 2 - In Progress label Sep 29, 2021

Fix subprogram marker line number

a46c22d

stuartarchibald added 6 commits October 26, 2021 11:38

Force code gen to look more like clang's output from C.

ef63bee

As title.

Pretend to be clang such that prologue treatment is correct

b443b2d

As title

Add test for break on symbol

fef9f3a

As title

flake8

d8a558e

as title

Fix unit test wrt new DWARF producer

bcfba3a

As title.

Correct line number

adb096e

As title

sklam approved these changes Oct 26, 2021

View reviewed changes

sklam requested changes Oct 26, 2021

View reviewed changes

sklam mentioned this pull request Oct 26, 2021

Fixup cuda debuginfo emission for 7177 #7514

Merged

sklam approved these changes Oct 27, 2021

View reviewed changes

sklam merged commit 944dcee into numba:master Oct 27, 2021

vlad-perevezentsev mentioned this pull request Nov 2, 2021

Incorrect result of info locals and info args commands when debugging w/ GDB #7527

Open

1 of 2 tasks

stuartarchibald mentioned this pull request Dec 15, 2021

Need to add more debugging tools - discuss #2623

Closed

PokhodenkoSA mentioned this pull request Dec 27, 2021

Access elements of a complex variable type (e.g. access different indexes of a multi-dimensional array) IntelPython/numba-dpex#484

Closed

This was referenced Jan 17, 2022

numba 0.55 breaks compatibility with coverage #7751

Closed

Fix #7751: Use original filename for array exprs #7752

Merged

Navigation Menu

Repository navigation

Add debug infomation support based on datamodel. - #7177

Add debug infomation support based on datamodel.#7177
sklam merged 32 commits into
numba:masterfrom
stuartarchibald:wip/dwarf_datamodel_1

stuartarchibald commented Jul 2, 2021

Uh oh!

stuartarchibald commented Sep 29, 2021

Uh oh!

stuartarchibald commented Oct 22, 2021

Uh oh!

sklam commented Oct 25, 2021

Uh oh!

stuartarchibald commented Oct 26, 2021

Uh oh!

sklam left a comment

Uh oh!

sklam commented Oct 26, 2021

Uh oh!

sklam left a comment

Uh oh!

gmarkall commented Oct 26, 2021

Uh oh!

sklam commented Oct 26, 2021

Uh oh!

sklam left a comment

Uh oh!

sklam commented Oct 27, 2021

Uh oh!

sklam commented Oct 27, 2021

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

stuartarchibald commented Jul 2, 2021

Uh oh!

stuartarchibald commented Sep 29, 2021

Uh oh!

stuartarchibald commented Oct 22, 2021

Uh oh!

sklam commented Oct 25, 2021

Uh oh!

stuartarchibald commented Oct 26, 2021

Uh oh!

sklam left a comment

Choose a reason for hiding this comment

Uh oh!

sklam commented Oct 26, 2021

Uh oh!

sklam left a comment

Choose a reason for hiding this comment

Uh oh!

gmarkall commented Oct 26, 2021

Uh oh!

sklam commented Oct 26, 2021

Uh oh!

sklam left a comment

Choose a reason for hiding this comment

Uh oh!

sklam commented Oct 27, 2021

Uh oh!

sklam commented Oct 27, 2021

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants