Skip to content

CI: Add GHA workflow to upload nightly wheels #22733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
QuLogic merged 2 commits into from
Mar 31, 2022
Merged

CI: Add GHA workflow to upload nightly wheels #22733

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6d3d6fd
CI: Add nightly upload of wheels to Anaconda Cloud
matthewfeickert Mar 30, 2022
7dda3c6
DOC: Add instructions on installing nightly build wheels
matthewfeickert Mar 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
CI: Add nightly upload of wheels to Anaconda Cloud 
Add nightlies.yml GitHub Action workflow that uses the gh CLI API to
download the 'wheels' artifact from the latest completed build of the
cibuildwheel workflow from the 'main' branch. This is done by getting
the fields 'event', 'status', and 'databaseId' from the output of
`gh run list` as JSON and then using jq to filter for 'push' events
(which correspond to merged PRs to the 'main' branch) that have
completed runs. This is then sorted by the databaseIds in descending
order to get the latest completed build run number. Then the 'wheels'
artifact can be downloaded.

The downloaded wheels are then uploaded to the scipy-wheels-nightly
Anaconda Cloud organization (https://anaconda.org/scipy-wheels-nightly)
using the anaconda-client CLI API.
  • Loading branch information
@matthewfeickert
matthewfeickert committed Mar 31, 2022
commit 6d3d6fdc71de72ff659c586ee8e363e41d0f3ef9
64 changes: 64 additions & 0 deletions .github/workflows/nightlies.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: Upload nightly wheels to Anaconda Cloud

on:
# Run daily at 1:23 UTC to upload nightly wheels to Anaconda Cloud
schedule:
- cron: '23 1 * * *'
# Run on demand with workflow dispatch
workflow_dispatch:

jobs:
upload_nightly_wheels:
name: Upload nightly wheels to Anaconda Cloud
runs-on: ubuntu-latest
Comment thread
matthewfeickert marked this conversation as resolved.
if: github.repository_owner == 'matplotlib'

steps:
- name: Install Python
uses: actions/setup-python@v3
with:
python-version: '3.x'
Comment thread
matthewfeickert marked this conversation as resolved.

# c.f. https://github.com/actions/download-artifact/issues/3#issuecomment-1017141067
- name: Download wheel artifacts from last build on 'main'
shell: bash
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
PROJECT_REPO="matplotlib/matplotlib"
BRANCH="main"
WORKFLOW_NAME="cibuildwheel.yml"
ARTIFACT_NAME="wheels"

gh run --repo "${PROJECT_REPO}" \
list --branch "${BRANCH}" \
--workflow "${WORKFLOW_NAME}" \
--json event,status,databaseId > runs.json
# Filter on 'push' events to main (merged PRs) that have completed
jq --compact-output \
'[ .[] | select(.event == "push") | select(.status == "completed") ]' \
runs.json > pushes.json
# Get id of latest build of wheels
RUN_ID=$(
jq --compact-output \
'sort_by(.databaseId) | reverse | .[0].databaseId' pushes.json
)
gh run --repo "${PROJECT_REPO}" \
download "${RUN_ID}" --name "${ARTIFACT_NAME}"

mkdir dist
mv *.whl dist/
ls -l dist/

- name: Install anaconda-client
run: |
python -m pip install --upgrade pip setuptools wheel
# c.f. https://github.com/Anaconda-Platform/anaconda-client/issues/540
python -m pip install git+https://github.com/Anaconda-Server/anaconda-client
python -m pip list

- name: Upload wheels to Anaconda Cloud as nightlies
run: |
anaconda --token ${{ secrets.ANACONDA_ORG_UPLOAD_TOKEN }} upload \

@matthewfeickert matthewfeickert Mar 30, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

c.f. #21637 (comment)

@tacaswell tacaswell Mar 31, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ANACONDA_ORG_UPLOAD_TOKEN is still the right secret but I do not think we will know if this works until we merge it.

@matthewfeickert matthewfeickert Mar 31, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it is worth it, but you could manually download a wheel and then upload it with the token to test in advance that the ANACONDA_ORG_UPLOAD_TOKEN is valid. Though given that this PR enables workflow dispatch if there was something wrong with the token you could just change the value of the repo secret and rerun the job (which runs in under 1 minute) so probably not worth the time to test it out before.

--user scipy-wheels-nightly \
dist/matplotlib-*.whl