Skip to content

Add build scripts for CUDA testing on gpuCI #7499

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sklam merged 11 commits into from
Oct 28, 2021
Merged

Add build scripts for CUDA testing on gpuCI #7499

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5a37c20
First pass at gpuCI build scripts
charlesbluca Oct 18, 2021
8845d7c
Don't need to test CUDA 9.0
charlesbluca Oct 18, 2021
96557e6
Make separate env for Numba testing
charlesbluca Oct 18, 2021
e8c7739
Apply suggestions from code review
charlesbluca Oct 19, 2021
512bdde
Fix environment creation
charlesbluca Oct 20, 2021
c53efdc
Use development version of llvmlite
charlesbluca Oct 20, 2021
15c636b
Document gpuCI
gmarkall Oct 21, 2021
2129c4e
README: Remove macOS from CUDA support list.
gmarkall Oct 27, 2021
af2da0c
build.sh: Align a hash in a comment
gmarkall Oct 27, 2021
1d2b545
gpuCI config changes based on PR #7499 feedback
gmarkall Oct 28, 2021
9d6d940
gpuCI build.sh: Remove dead hasArg code
gmarkall Oct 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ Supported Platforms

* (Optional) Accelerators and GPUs:

* NVIDIA GPUs (Kepler architecture or later) via CUDA driver on Linux,
Windows, macOS (< 10.14).
* NVIDIA GPUs (Kepler architecture or later) via CUDA driver on Linux and
Windows.

Dependencies
============
Expand Down
21 changes: 21 additions & 0 deletions buildscripts/gpuci/axis.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
PYTHON_VER:
- "3.8"

CUDA_VER:
- "11.2"

CUDA_TOOLKIT_VER:
- "9.2"
- "10.0"
- "10.2"
- "11.0"
- "11.2"
- "11.4"

LINUX_VER:
- ubuntu18.04

RAPIDS_VER:
- "21.12"
Comment thread
stuartarchibald marked this conversation as resolved.
Resolved

excludes:

@esc esc Oct 27, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick: what does this do, is it needed?

@gmarkall gmarkall Oct 27, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file defines a matrix of configurations that is the cross-product of all the axes, excludes can be used to remove some configurations from the generated set of configurations, so you don't have to run the cross-product, but only the subset you're interested in. This will probably get a few entries once i start making changes to the config to test the NVIDIA conda packages and the NVIDIA CUDA Python bindings with Numba.

It is documented at: https://github.com/jenkinsci/yaml-axis-plugin/blob/master/README.md#excluding-logic

64 changes: 64 additions & 0 deletions buildscripts/gpuci/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
##############################################
# Numba GPU build and test script for CI #

@esc esc Oct 27, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the final hash # symbol appears to be misaligned.

@gmarkall gmarkall Oct 27, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now aligned.

##############################################
set -e
NUMARGS=$#
ARGS=$*

# Arg parsing function
function hasArg {
(( ${NUMARGS} != 0 )) && (echo " ${ARGS} " | grep -q " $1 ")
}

@stuartarchibald stuartarchibald Oct 27, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dead code?

@gmarkall gmarkall Oct 28, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably, will try removing.


# Set path and build parallel level
export PATH=/opt/conda/bin:/usr/local/cuda/bin:$PATH
export PARALLEL_LEVEL=${PARALLEL_LEVEL:-4}

# Set home to the job's workspace
export HOME="$WORKSPACE"

# Switch to project root; also root of repo checkout
cd "$WORKSPACE"

# Determine CUDA release version
export CUDA_REL=${CUDA_VERSION%.*}

################################################################################
# SETUP - Check environment
################################################################################

gpuci_logger "Check environment variables"
env

gpuci_logger "Check GPU usage"
nvidia-smi

gpuci_logger "Create testing env"
. /opt/conda/etc/profile.d/conda.sh
gpuci_mamba_retry create -n numba -y \

@stuartarchibald stuartarchibald Oct 27, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
gpuci_mamba_retry create -n numba -y \
gpuci_mamba_retry create -n numba_ci -y \

suggest using a different env name to the package name as it can help with debugging.

@gmarkall gmarkall Oct 28, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point.

"python=${PYTHON_VER}" \
"cudatoolkit=${CUDA_TOOLKIT_VER}" \

@stuartarchibald stuartarchibald Oct 27, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the Anaconda distro compilers should ideally be used here such that the gpuCI build is testing issues arising solely from CUDA and not through the use of different compilers. Think the Anaconda compiler packages are gcc_linux-64=7 and gxx_linux-64=7 (pin to 7 at present for libc etc compat).

@gmarkall gmarkall Oct 28, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WIll add the Anaconda compilers.

"numba/label/dev::llvmlite" \
"numpy" \
"scipy" \
"cffi"
Comment thread
stuartarchibald marked this conversation as resolved.
Resolved
conda activate numba

@stuartarchibald stuartarchibald Oct 27, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
conda activate numba
conda activate numba_ci

for consistency with the above suggestion.

@gmarkall gmarkall Oct 28, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will change as above.


gpuci_logger "Install numba"
python setup.py develop

gpuci_logger "Check Python version"
python --version
$CC --version
$CXX --version

@stuartarchibald stuartarchibald Oct 27, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can probably remove this if the compilers are set from Anaconda distro. Also numba -s (below) dumps python info.

@gmarkall gmarkall Oct 28, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll remove the Python info, but I'll leave in the compiler checks in case some oddity results in the Anaconda compilers being installed but not used (e.g. in case there's something about the image we run on that is / becomes unanticipated).


gpuci_logger "Check conda environment"
conda info
conda config --show-sources
conda list --show-channel-urls

@stuartarchibald stuartarchibald Oct 27, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this covered by numba -s?

@gmarkall gmarkall Oct 28, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.


gpuci_logger "Dump system information from Numba"
python -m numba -s

gpuci_logger "Run tests in numba.cuda.tests"
python -m numba.runtests numba.cuda.tests -m
12 changes: 10 additions & 2 deletions docs/source/developer/contributing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,8 +329,16 @@ As a result of the above, public CI is implemented as follows:
matrix and each combination runs one chunk. This is done for speed,
because public CI cannot cope with the load else.

If a pull request is changing CUDA code (which cannot be tested on Public CI as
there's no hardware) or it is making changes to something that the core
If a Pull Request (PR) changes CUDA code or will affect the CUDA target, it
needs to be run on `gpuCI <https://gpuci.gpuopenanalytics.com/job/numba/>`_.
This can be triggered by one of the Numba maintainers commenting ``run gpuCI
tests`` on the PR discussion. This runs the CUDA testsuite with various CUDA
toolkit versions on Linux, to provide some initial confidence in the
correctness of the changes with respect to CUDA. Following approval, the PR
will also be run on Numba's build farm to test other configurations with CUDA
(including Windows, which is not tested by gpuCI).

If the PR is not CUDA-related but makes changes to something that the core
developers consider risky, then it will also be run on the Numba farm just to
make sure. The Numba project's private build and test farm will actually
exercise all the applicable tests on all the combinations noted above on real
Expand Down