Add build scripts for CUDA testing on gpuCI by charlesbluca · Pull Request #7499 · numba/numba

charlesbluca · 2021-10-20T15:25:13Z

This PR adds the relevant build scripts to run Numba's CUDA tests on gpuCI for pull requests. Some key things to note:

gpuCI tests will not be run by default on PRs; instead they are triggered with a phrase commented on the PR by the following gpuCI admins for this repo:

gmarkall
testhound
stuartarchibald
sklam
esc
charlesbluca
quasiben

currently the phrase we're considering is gpuci run tests, happy to discuss that more in this issue though
tests will be ran with Python 3.8 and CUDA toolkit 9.2, 10.0, 10.2, 11.0, 11.2, and 11.4; this can be changed by modifying the axis.yaml added by this PR accordingly
tests are ran in the gpuci/rapidsai docker container, with a separate environment being created with Numba's build requirements; in the future, we should be able to move to custom Numba images with all requirements preinstalled if timing is an issue (note that this is why we specify a RAPIDS_VER and CUDA_VER in axis.yaml, even though these go unused for the testing)
currently Numba does not support outputting these test results in JUnit XML format (to my knowledge); if this functionality is added, we can modify these scripts so that the results of each individual gpuCI job (i.e. cudatoolkit=9.2, 10.0, etc.) are posted to the triggering PR

Once this PR is merged in, we can do the internal work to target gpuCI to this repo, and verify that things are working properly with a follow-up test PR; for an example of the tests in action, view charlesbluca#1

Reference an existing issue

This is a second attempt at #5123.

Co-authored-by: Graham Markall <535640+gmarkall@users.noreply.github.com>

gmarkall · 2021-10-21T11:03:33Z

I've added some docs that explain how we can trigger the CI. At this point I think the PR / setup is ready for discussion with @esc @stuartarchibald @sklam.

gmarkall · 2021-10-27T08:54:03Z

gpuci run tests

esc · 2021-10-27T09:32:36Z

We are currently stuck on getting GPUCI to report it's status back to Github.

gmarkall · 2021-10-27T09:39:58Z

I've got it to report manually using a token I generated myself with the repo:status scope only - I'll see if I can add this to the gpuCI config, and then there should be no further action needed on the Numba repo side.

gmarkall · 2021-10-27T14:39:38Z

gpuci rerun tests

gmarkall · 2021-10-27T14:41:03Z

gpuci run tests

gmarkall · 2021-10-27T14:46:56Z

gpuci run tests

gmarkall · 2021-10-27T14:50:46Z

gpuci run tests

gmarkall · 2021-10-27T15:11:58Z

@esc We now have the check status updating automatically from gpuCI! I added another commit onto this PR that fixes a tiny doc issue mentioned in #7513 so that we had a "fresh" commit to test with.

I think this is now ready for any final review / merging.

esc

Some minor points to discuss, but otherwise ready to merge.

@gmarkall thank you so much for setting this up. Having a public CI with GPU enabled will be a real boon! 🌸

esc · 2021-10-27T16:13:39Z

+RAPIDS_VER:
+- "21.12"
+
+excludes:


Nitpick: what does this do, is it needed?

This file defines a matrix of configurations that is the cross-product of all the axes, excludes can be used to remove some configurations from the generated set of configurations, so you don't have to run the cross-product, but only the subset you're interested in. This will probably get a few entries once i start making changes to the config to test the NVIDIA conda packages and the NVIDIA CUDA Python bindings with Numba.

It is documented at: https://github.com/jenkinsci/yaml-axis-plugin/blob/master/README.md#excluding-logic

esc · 2021-10-27T16:14:08Z

@@ -0,0 +1,64 @@
+##############################################
+# Numba GPU build and test script for CI      #


the final hash # symbol appears to be misaligned.

This is now aligned.

No functional change, just to improve the aesthetics of the script.

gmarkall · 2021-10-27T16:36:12Z

@esc Many thanks for the review! Comments added on discussion points.

stuartarchibald

Thanks for the patch and for setting this up. It's great to see Numba having this integration as it means that PRs can be tested by more people without reliance on the build farm (which is often under pressure near releases/needs to be maintained at times!). I've a few suggestions, otherwise looks good! Thanks again!

stuartarchibald · 2021-10-27T16:30:34Z

+
+gpuci_logger "Create testing env"
+. /opt/conda/etc/profile.d/conda.sh
+gpuci_mamba_retry create -n numba -y \


Suggested change

gpuci_mamba_retry create -n numba -y \

gpuci_mamba_retry create -n numba_ci -y \

suggest using a different env name to the package name as it can help with debugging.

Good point.

stuartarchibald · 2021-10-27T16:30:51Z

+                  "numpy" \
+                  "scipy" \
+                  "cffi" 
+conda activate numba


Suggested change

conda activate numba

conda activate numba_ci

for consistency with the above suggestion.

Will change as above.

stuartarchibald · 2021-10-27T16:34:09Z

+. /opt/conda/etc/profile.d/conda.sh
+gpuci_mamba_retry create -n numba -y \
+                  "python=${PYTHON_VER}" \
+                  "cudatoolkit=${CUDA_TOOLKIT_VER}" \


I think the Anaconda distro compilers should ideally be used here such that the gpuCI build is testing issues arising solely from CUDA and not through the use of different compilers. Think the Anaconda compiler packages are gcc_linux-64=7 and gxx_linux-64=7 (pin to 7 at present for libc etc compat).

WIll add the Anaconda compilers.

stuartarchibald · 2021-10-27T16:35:03Z

+gpuci_logger "Check Python version"
+python --version
+$CC --version
+$CXX --version


Can probably remove this if the compilers are set from Anaconda distro. Also numba -s (below) dumps python info.

I'll remove the Python info, but I'll leave in the compiler checks in case some oddity results in the Anaconda compilers being installed but not used (e.g. in case there's something about the image we run on that is / becomes unanticipated).

stuartarchibald · 2021-10-27T16:36:25Z

+gpuci_logger "Check conda environment"
+conda info
+conda config --show-sources
+conda list --show-channel-urls


Is this covered by numba -s?

stuartarchibald · 2021-10-27T16:37:48Z

+NUMARGS=$#
+ARGS=$*
+
+# Arg parsing function
+function hasArg {
+    (( ${NUMARGS} != 0 )) && (echo " ${ARGS} " | grep -q " $1 ")
+}


Probably, will try removing.

- Rename conda env to numba_ci so it's not "numba", which will be less confusing (e.g. reading paths). - Use Anaconda compilers. - Remove Python version check, which comes from numba -s also. - Remove conda list, which comes from numba -s also A couple of other changes not based on the feedback: - Install psutil, suggested by numba -s - Run in verbose mode, so if there's a crash in a test we can see which one.

gmarkall · 2021-10-28T10:19:56Z

gpuci run tests

gmarkall · 2021-10-28T10:35:53Z

@stuartarchibald Thanks for the review! I believe all comments are addressed (and gpuCI still passes).

stuartarchibald

Patch looks good! Thanks for the implementation, setup and fixes @charlesbluca and @gmarkall!

stuartarchibald · 2021-10-28T10:48:48Z

@charlesbluca Congratulations on your first contribution to Numba!

* python3.10: (61 commits) fix branch pruner tests alternative error for unsupported predicates for 3.10 enable peephole for list_to_tuple in 3.10 implement GEN_START bytecode instruction for 3.10 fix unroller defeat 3.10 optimizer fix polarity of offset correction Fixup instruction pointer offsets fix cpython tracing API bump max python version gpuCI build.sh: Remove dead hasArg code gpuCI config changes based on PR numba#7499 feedback build.sh: Align a hash in a comment More fixes Fix missing arg to _add_subprogram() README: Remove macOS from CUDA support list. Fixup for NvvmDIBuilder Correct line number Fix unit test wrt new DWARF producer flake8 Add test for break on symbol ...

charlesbluca and others added 6 commits October 18, 2021 10:38

First pass at gpuCI build scripts

5a37c20

Don't need to test CUDA 9.0

8845d7c

Make separate env for Numba testing

96557e6

Apply suggestions from code review

e8c7739

Co-authored-by: Graham Markall <535640+gmarkall@users.noreply.github.com>

Fix environment creation

512bdde

Use development version of llvmlite

c53efdc

Co-authored-by: Graham Markall <535640+gmarkall@users.noreply.github.com>

charlesbluca requested review from esc, sklam and stuartarchibald as code owners October 20, 2021 15:25

gmarkall added 2 - In Progress CUDA CUDA related issue/PR labels Oct 20, 2021

Document gpuCI

15c636b

gmarkall added 3 - Ready for Review Effort - medium Medium size effort needed and removed 2 - In Progress labels Oct 21, 2021

README: Remove macOS from CUDA support list.

2129c4e

gmarkall added 4 - Waiting on author Waiting for author to respond to review and removed 3 - Ready for Review labels Oct 27, 2021

esc requested changes Oct 27, 2021

View reviewed changes

build.sh: Align a hash in a comment

af2da0c

No functional change, just to improve the aesthetics of the script.

gmarkall added 4 - Waiting on reviewer Waiting for reviewer to respond to author and removed 4 - Waiting on author Waiting for author to respond to review labels Oct 27, 2021

stuartarchibald reviewed Oct 27, 2021

View reviewed changes

gmarkall added 2 commits October 28, 2021 11:13

gpuCI build.sh: Remove dead hasArg code

9d6d940

stuartarchibald approved these changes Oct 28, 2021

View reviewed changes

stuartarchibald added 5 - Ready to merge Review and testing done, is ready to merge and removed 4 - Waiting on reviewer Waiting for reviewer to respond to author labels Oct 28, 2021

stuartarchibald added this to the Numba 0.55 RC milestone Oct 28, 2021

esc approved these changes Oct 28, 2021

View reviewed changes

sklam merged commit a7cc0d0 into numba:master Oct 28, 2021

		@@ -0,0 +1,64 @@
		##############################################
		# Numba GPU build and test script for CI #

	gpuci_mamba_retry create -n numba -y \
	gpuci_mamba_retry create -n numba_ci -y \

Conversation

charlesbluca commented Oct 20, 2021

Reference an existing issue

Uh oh!

gmarkall commented Oct 21, 2021

Uh oh!

gmarkall commented Oct 27, 2021

Uh oh!

esc commented Oct 27, 2021

Uh oh!

gmarkall commented Oct 27, 2021

Uh oh!

gmarkall commented Oct 27, 2021

Uh oh!

gmarkall commented Oct 27, 2021

Uh oh!

gmarkall commented Oct 27, 2021

Uh oh!

gmarkall commented Oct 27, 2021

Uh oh!

gmarkall commented Oct 27, 2021

Uh oh!

esc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmarkall commented Oct 27, 2021

Uh oh!

stuartarchibald left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmarkall commented Oct 28, 2021

Uh oh!

gmarkall commented Oct 28, 2021

Uh oh!

stuartarchibald left a comment

Choose a reason for hiding this comment

Uh oh!

stuartarchibald commented Oct 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!