Skip to content

Commit 42616dc

Browse files
sklamsklam
authored
Merge pull request numba#9436 from sklam/misc/tests_timing_and_xml
Add timing and junit xml output to testsuite
2 parents a7fc4b5 + 2d5b346 commit 42616dc

8 files changed

Lines changed: 322 additions & 40 deletions

File tree

‎buildscripts/azure/azure-linux-macos.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,3 +49,10 @@ jobs:
4949
export PATH=$HOME/miniconda3/bin:$PATH
5050
buildscripts/incremental/test.sh
5151
displayName: 'Test'
52+
53+
- task: PublishTestResults@2
54+
condition: succeededOrFailed()
55+
displayName: Publish Test Results
56+
inputs:
57+
testResultsFiles: 'junit_numba_*.xml'
58+
mergeTestResults: true

‎buildscripts/azure/azure-windows.yml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,5 +60,12 @@ jobs:
6060
set NUMBA_CAPTURED_ERRORS=new_style
6161
set NUMBA_ENABLE_CUDASIM=1
6262
echo "Running shard of discovered tests: %TEST_START_INDEX%:%TEST_COUNT%"
63-
python runtests.py -m 2 -b -j "%TEST_START_INDEX%:%TEST_COUNT%" --exclude-tags='long_running' -- numba.tests
63+
python runtests.py -m 2 -b -j "%TEST_START_INDEX%:%TEST_COUNT%" --exclude-tags='long_running' --junit -- numba.tests
6464
displayName: 'Test shard of test files'
65+
66+
- task: PublishTestResults@2
67+
condition: succeededOrFailed()
68+
displayName: Publish Test Results
69+
inputs:
70+
testResultsFiles: 'junit_numba_*.xml'
71+
mergeTestResults: true

‎buildscripts/incremental/test.sh‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -136,16 +136,16 @@ echo "INFO: Running shard of discovered tests: ($TEST_START_INDEX:$TEST_COUNT)"
136136
if [ "$RUN_COVERAGE" == "yes" ]; then
137137
export PYTHONPATH=.
138138
coverage erase
139-
$SEGVCATCH coverage run runtests.py -b -j "$TEST_START_INDEX:$TEST_COUNT" --exclude-tags='long_running' -m $TEST_NPROCS -- numba.tests
139+
$SEGVCATCH coverage run runtests.py -b -j "$TEST_START_INDEX:$TEST_COUNT" --exclude-tags='long_running' -m $TEST_NPROCS --junit -- numba.tests
140140
elif [ "$RUN_TYPEGUARD" == "yes" ]; then
141141
echo "INFO: Running with typeguard"
142-
NUMBA_USE_TYPEGUARD=1 NUMBA_ENABLE_CUDASIM=1 PYTHONWARNINGS="ignore:::typeguard" $SEGVCATCH python runtests.py -b -j "$TEST_START_INDEX:$TEST_COUNT" --exclude-tags='long_running' -m $TEST_NPROCS -- numba.tests
142+
NUMBA_USE_TYPEGUARD=1 NUMBA_ENABLE_CUDASIM=1 PYTHONWARNINGS="ignore:::typeguard" $SEGVCATCH python runtests.py -b -j "$TEST_START_INDEX:$TEST_COUNT" --exclude-tags='long_running' -m $TEST_NPROCS --junit -- numba.tests
143143
else
144-
NUMBA_ENABLE_CUDASIM=1 $SEGVCATCH python -m numba.runtests -b -j "$TEST_START_INDEX:$TEST_COUNT" --exclude-tags='long_running' -m $TEST_NPROCS -- numba.tests
144+
NUMBA_ENABLE_CUDASIM=1 $SEGVCATCH python -m numba.runtests -b -j "$TEST_START_INDEX:$TEST_COUNT" --exclude-tags='long_running' -m $TEST_NPROCS --junit -- numba.tests
145145
fi
146146

147147
# Now run with RVSDG enabled for a subset of tests
148148
if [[ "$TEST_RVSDG" == "yes" ]]; then
149149
echo "Running RVSDG tests..."
150-
NUMBA_USE_RVSDG_FRONTEND=1 NUMBA_CAPTURED_ERRORS=new_style NUMBA_ENABLE_CUDASIM=1 $SEGVCATCH python -m numba.runtests -b -v -m $TEST_NPROCS -- numba.tests.test_usecases
150+
NUMBA_USE_RVSDG_FRONTEND=1 NUMBA_CAPTURED_ERRORS=new_style NUMBA_ENABLE_CUDASIM=1 $SEGVCATCH python -m numba.runtests -b -v -m $TEST_NPROCS --junit -- numba.tests.test_usecases
151151
fi

‎docs/source/developer/contributing.rst‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -214,6 +214,9 @@ instead. For example::
214214
$ NUMBA_USE_TYPEGUARD=1 python runtests.py
215215

216216

217+
See also: :ref:`Debugging the Test Suite <developer-testsuite>`
218+
219+
217220
Running coverage
218221
''''''''''''''''
219222

‎docs/source/developer/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ Developer Manual
2727
literal.rst
2828
llvm_timings.rst
2929
debugging.rst
30+
testsuite.rst
3031
event_api.rst
3132
target_extension.rst
3233
bytecode.rst

‎docs/source/developer/testsuite.rst‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
.. _developer-testsuite:
2+
3+
==============================
4+
Debugging the Numba Test Suite
5+
==============================
6+
7+
.. note:: Features described here are only implemented for the parallel
8+
test runner (e.g. requires the ``-m`` flag).
9+
10+
The Numba test suite provides some useful features for debugging test failures
11+
or crashes.
12+
13+
Generating JUnit XML Output
14+
---------------------------
15+
16+
You can generate a JUnit-compatible XML report by running the tests with the
17+
``--junit`` flag:
18+
19+
.. code-block:: bash
20+
21+
$ python -m numba.runtests <args> --junit
22+
23+
This will produce a ``junit_numba_<timestamp>.xml`` file containing the test results.
24+
25+
The XML contains some extra metadata as ``testcase`` properties:
26+
27+
- ``pid``: The ID of the worker process running each test case
28+
- ``start_time``: The start time for each test case relative to when the worker
29+
process started
30+
31+
32+
Example of the XML file:
33+
34+
.. code-block:: xml
35+
36+
<?xml version='1.0' encoding='utf-8'?>
37+
<testsuites time="20.745398832">
38+
<testsuite name="numba_testsuite">
39+
<testcase name="TestUsecases.test_andor" classname="numba.tests.test_usecases"
40+
time="1.7033392500000002">
41+
<properties>
42+
<property name="pid" value="73274" />
43+
<property name="start_time" value="1.337771667" />
44+
</properties>
45+
</testcase>
46+
<testcase name="TestUsecases.test_blackscholes_cnd" classname="numba.tests.test_usecases"
47+
time="1.723800292">
48+
<properties>
49+
<property name="pid" value="73275" />
50+
<property name="start_time" value="1.3350285" />
51+
</properties>
52+
</testcase>
53+
...
54+
</testsuite>
55+
</testsuites>
56+
57+
58+
Debugging Timeouts caused by Crashes
59+
------------------------------------
60+
61+
If the test suite times out, it may indicate a crashed worker process.
62+
In this case, the timeout error will show:
63+
64+
- The active tests that didn't finish before the timeout
65+
- For each affected process ID, the list of tests that process was running
66+
67+
For example:
68+
69+
.. code-block:: none
70+
71+
TimeoutError: Active tests didn't finish before timeout:
72+
- test_foo [PID 1234]
73+
74+
Tests ran by each affected process:
75+
- [PID 1234]: test_bar test_baz test_foo
76+
77+
This information can help reproduce crashes that depend on the test execution
78+
order or state within a process. You can retry just the tests that were running
79+
in the crashed process.
80+

0 commit comments

Comments
 (0)