Skip to content

Commit 8861c0c

Browse files
lafionakevingurney
andauthored
ARROW-15691: [Dev] Update archery to work with either master or main as default branch (apache#14033)
# Overview The goal of this pull request is to update `archery` to work with a repository default branch named `master` or `main`, as part of the effort to rename the Apache Arrow repository's default branch to `main`. The parent Jira ticket can be found [here](https://issues.apache.org/jira/browse/ARROW-15689). # Implementation - Update the language of the top level `archery`, `crossbow`, and `docker` command line interface code to reference the mainline development branch (default git branch) generically. - Update comments that reference the `master` branch. - Update the `crossbow` benchmarking examples to generically specify the `<default-branch>` rather than a hard-coded value. - In `.github/workflows/integration.yml`, add an environment variable `DEFAULT_BRANCH` to the `archery` command in the "Execute Docker Build" step, so that `archery` can reliably access the default branch value. - In `.github/workflows/archery.yml`, add an environment variable `DEFAULT_BRANCH` for all steps. This environment variable was already used by the `Git Fixup` step. It will also be used by the `Archery Unittests` step. - Add a property, `default_branch_name`, to the `Repo` class in `dev/archery/archery/crossbow/core.py` for computing the default branch name. - If specified, the `DEFAULT_BRANCH` environment variable, takes precedent in determining the default branch name (this is for overriding the git-based heuristic and qualifying in CI). - Otherwise, `pygit2` is used to get the default branch name via the Apache Arrow repository's `origin` remote `HEAD` reference. This is a heuristic, but in most cases, the `HEAD` reference of the remote points to the default branch. - Add a cached property, default_branch to the `Release` class in `dev/archery/archery/release/core.py` for computing the default branch name. Similar to the `default_branch_name` property for `Repo` in `archery/archery/crossbow/core.py`: - If specified, the `DEFAULT_BRANCH` environment variable, takes precedent in determining the default branch name (this is for qualifying in CI). - Otherwise, similar to the previous step,`GitPython` is used to get the default branch name via the Apache Arrow repository's `origin` remote `HEAD` reference. - Modify the `PANDAS` and `DASK` Docker Build Parameter value for indicating the upstream development branch to `upstream_devel`. - Updated Development Running Docker Builds documentation to reflect the above change and fixed a broken link. #### Out of scope: - There are remaining instances of `master` in the test fixtures files in `dev/archery/archery/test/fixtures`. It appears that the data only refers to external repositories, such as `ursa-labs/ursabot`, which currently uses `master`, so these instances were not modified. # Testing - Ran the `archery` and `crossbow` commands in local clones of both the `mathworks/arrow` and `apache/arrow` repositories. - Confirmed that the GitHub CI jobs pass. - We are unsure how to locally qualify the changes to the `release` component, but the `release` tests pass in CI. - Ran a sample `archery docker` command after setting the `PANDAS` environment variable to confirm that the correct version of Pandas is used. # Future Directions 1. Added Jira task to update the pull request merge script to work with both `master` and `main` ([ARROW-17777](https://issues.apache.org/jira/browse/ARROW-17777)) 2. Added Jira task to update the default value for the default branch name, if it cannot be determined via an environment variable, `ARCHERY_DEFAULT_BRANCH`, or from the repository's remote head reference. ([ARROW-18011](https://issues.apache.org/jira/browse/ARROW-18011)) # Notes Thank you @ kevingurney for your help with this pull request! Lead-authored-by: Fiona La <fionala7@gmail.com> Co-authored-by: Kevin Gurney <kgurney@mathworks.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
1 parent 21564cf commit 8861c0c

13 files changed

Lines changed: 122 additions & 33 deletions

File tree

‎.github/workflows/archery.yml‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ on:
3131
- 'dev/tasks/**'
3232
- 'docker-compose.yml'
3333

34+
env:
35+
ARCHERY_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
36+
3437
concurrency:
3538
group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
3639
cancel-in-progress: true
@@ -52,9 +55,7 @@ jobs:
5255
fetch-depth: 0
5356
- name: Git Fixup
5457
shell: bash
55-
run: |
56-
DEFAULT_BRANCH=${{ github.event.repository.default_branch }}
57-
git branch $DEFAULT_BRANCH origin/$DEFAULT_BRANCH || true
58+
run: git branch $ARCHERY_DEFAULT_BRANCH origin/$ARCHERY_DEFAULT_BRANCH || true
5859
- name: Setup Python
5960
uses: actions/setup-python@v4
6061
with:

‎.github/workflows/integration.yml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,11 @@ jobs:
8585
env:
8686
ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
8787
ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
88-
run: archery docker run -e ARCHERY_INTEGRATION_WITH_RUST=1 conda-integration
88+
run: >
89+
archery docker run \
90+
-e ARCHERY_DEFAULT_BRANCH=${{ github.event.repository.default_branch }} \
91+
-e ARCHERY_INTEGRATION_WITH_RUST=1 \
92+
conda-integration
8993
- name: Docker Push
9094
if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
9195
env:

‎.travis.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,7 @@ install:
184184
- sudo -H pip3 install -e dev/archery[docker]
185185

186186
script:
187+
- export ARCHERY_DEFAULT_BRANCH=$(git rev-parse --abbrev-ref origin/HEAD | sed s@origin/@@)
187188
- |
188189
archery docker run \
189190
${DOCKER_RUN_ARGS} \

‎ci/scripts/install_dask.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ fi
2626

2727
dask=$1
2828

29-
if [ "${dask}" = "master" ]; then
29+
if [ "${dask}" = "upstream_devel" ]; then
3030
pip install https://github.com/dask/dask/archive/main.tar.gz#egg=dask[dataframe]
3131
elif [ "${dask}" = "latest" ]; then
3232
pip install dask[dataframe]

‎ci/scripts/install_pandas.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ else
3535
pip install numpy==${numpy}
3636
fi
3737

38-
if [ "${pandas}" = "master" ]; then
38+
if [ "${pandas}" = "upstream_devel" ]; then
3939
pip install git+https://github.com/pandas-dev/pandas.git --no-build-isolation
4040
elif [ "${pandas}" = "nightly" ]; then
4141
pip install --extra-index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple --pre pandas

‎dev/archery/archery/cli.py‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -529,7 +529,7 @@ def benchmark_run(ctx, rev_or_path, src, preserve, output, cmake_extras,
529529
help="Hide counters field in diff report.")
530530
@click.argument("contender", metavar="[<contender>",
531531
default=ArrowSources.WORKSPACE, required=False)
532-
@click.argument("baseline", metavar="[<baseline>]]", default="origin/master",
532+
@click.argument("baseline", metavar="[<baseline>]]", default="origin/HEAD",
533533
required=False)
534534
@click.pass_context
535535
def benchmark_diff(ctx, src, preserve, output, language, cmake_extras,
@@ -542,7 +542,8 @@ def benchmark_diff(ctx, src, preserve, output, language, cmake_extras,
542542
543543
The caller can optionally specify both the contender and the baseline. If
544544
unspecified, the contender will default to the current workspace (like git)
545-
and the baseline will default to master.
545+
and the baseline will default to the mainline development branch (i.e.
546+
default git branch).
546547
547548
Each target (contender or baseline) can either be a git revision
548549
(commit, tag, special values like HEAD) or a cmake build directory. This
@@ -559,16 +560,18 @@ def benchmark_diff(ctx, src, preserve, output, language, cmake_extras,
559560
Examples:
560561
561562
\b
562-
# Compare workspace (contender) with master (baseline)
563+
# Compare workspace (contender) against the mainline development branch
564+
# (baseline)
563565
\b
564566
archery benchmark diff
565567
566568
\b
567-
# Compare master (contender) with latest version (baseline)
569+
# Compare the mainline development branch (contender) against the latest
570+
# version (baseline)
568571
\b
569572
export LAST=$(git tag -l "apache-arrow-[0-9]*" | sort -rV | head -1)
570573
\b
571-
archery benchmark diff master "$LAST"
574+
archery benchmark diff <default-branch> "$LAST"
572575
573576
\b
574577
# Compare g++7 (contender) with clang++-8 (baseline) builds

‎dev/archery/archery/crossbow/cli.py‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,7 @@ def check_config(obj, config_path):
9696
'locally. Examples: https://github.com/apache/arrow or '
9797
'https://github.com/kszucs/arrow.')
9898
@click.option('--arrow-branch', '-b', default=None,
99-
help='Give the branch name explicitly, e.g. master, ARROW-1949.')
99+
help='Give the branch name explicitly, e.g. ARROW-1949.')
100100
@click.option('--arrow-sha', '-t', default=None,
101101
help='Set commit SHA or Tag name explicitly, e.g. f67a515, '
102102
'apache-arrow-0.11.1.')
@@ -157,7 +157,7 @@ def submit(obj, tasks, groups, params, job_prefix, config_path, arrow_version,
157157

158158

159159
@crossbow.command()
160-
@click.option('--base-branch', default="master",
160+
@click.option('--base-branch', default=None,
161161
help='Set base branch for the PR.')
162162
@click.option('--create-pr', is_flag=True, default=False,
163163
help='Create GitHub Pull Request')
@@ -192,6 +192,12 @@ def verify_release_candidate(obj, base_branch, create_pr,
192192

193193
# Redefine Arrow repo to use the correct arrow remote.
194194
arrow = Repo(path=obj['arrow'].path, remote_url=remote)
195+
196+
# Default value for base_branch is the repository's default branch name
197+
if base_branch is None:
198+
# Get the default branch name from the repository
199+
base_branch = arrow.default_branch_name
200+
195201
response = arrow.github_pr(title=pr_title, head=head_branch,
196202
base=base_branch, body=pr_body,
197203
github_token=obj['queue'].github_token,
@@ -225,7 +231,7 @@ def verify_release_candidate(obj, base_branch, create_pr,
225231
'locally. Examples: https://github.com/apache/arrow or '
226232
'https://github.com/kszucs/arrow.')
227233
@click.option('--arrow-branch', '-b', default=None,
228-
help='Give the branch name explicitly, e.g. master, ARROW-1949.')
234+
help='Give the branch name explicitly, e.g. ARROW-1949.')
229235
@click.option('--arrow-sha', '-t', default=None,
230236
help='Set commit SHA or Tag name explicitly, e.g. f67a515, '
231237
'apache-arrow-0.11.1.')

‎dev/archery/archery/crossbow/core.py‎

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
from io import StringIO
2929
from pathlib import Path
3030
from datetime import date
31+
import warnings
3132

3233
import jinja2
3334
from ruamel.yaml import YAML
@@ -133,7 +134,7 @@ def format_all(items, pattern):
133134

134135
# configurations for setting up branch skipping
135136
# - appveyor has a feature to skip builds without an appveyor.yml
136-
# - travis reads from the master branch and applies the rules
137+
# - travis reads from the default branch and applies the rules
137138
# - circle requires the configuration to be present on all branch, even ones
138139
# that are configured to be skipped
139140
# - azure skips branches without azure-pipelines.yml by default
@@ -361,6 +362,29 @@ def signature(self):
361362
return pygit2.Signature(self.user_name, self.user_email,
362363
int(time.time()))
363364

365+
@property
366+
def default_branch_name(self):
367+
default_branch_name = os.getenv("ARCHERY_DEFAULT_BRANCH")
368+
369+
if default_branch_name is None:
370+
try:
371+
ref_obj = self.repo.references["refs/remotes/origin/HEAD"]
372+
target_name = ref_obj.target
373+
target_name_tokenized = target_name.split("/")
374+
default_branch_name = target_name_tokenized[-1]
375+
except KeyError:
376+
# TODO: ARROW-18011 to track changing the hard coded default
377+
# value from "master" to "main".
378+
default_branch_name = "master"
379+
warnings.warn('Unable to determine default branch name: '
380+
'ARCHERY_DEFAULT_BRANCH environment variable is '
381+
'not set. Git repository does not contain a '
382+
'\'refs/remotes/origin/HEAD\'reference. Setting '
383+
'the default branch name to ' +
384+
default_branch_name, RuntimeWarning)
385+
386+
return default_branch_name
387+
364388
def create_tree(self, files):
365389
builder = self.repo.TreeBuilder()
366390

@@ -382,7 +406,7 @@ def create_commit(self, files, parents=None, message='',
382406
if parents is None:
383407
# by default use the main branch as the base of the new branch
384408
# required to reuse github actions cache across crossbow tasks
385-
commit, _ = self.repo.resolve_refish("master")
409+
commit, _ = self.repo.resolve_refish(self.default_branch_name)
386410
parents = [commit.id]
387411
tree_id = self.create_tree(files)
388412

@@ -546,8 +570,10 @@ def github_overwrite_release_assets(self, tag_name, target_commitish,
546570
'Unsupported upload method {}'.format(method)
547571
)
548572

549-
def github_pr(self, title, head=None, base="master", body=None,
573+
def github_pr(self, title, head=None, base=None, body=None,
550574
github_token=None, create=False):
575+
# Default value for base is the default_branch_name
576+
base = self.default_branch_name if base is None else base
551577
github_token = github_token or self.github_token
552578
repo = self.as_github_repo(github_token=github_token)
553579
if create:
@@ -1289,11 +1315,15 @@ def validate(self):
12891315
'is: `{}`'.format(task_name, str(e))
12901316
)
12911317

1318+
# Get the default branch name from the repository
1319+
arrow_source_dir = ArrowSources.find()
1320+
repo = Repo(arrow_source_dir.path)
1321+
12921322
# validate that the defined tasks are renderable, in order to to that
12931323
# define the required object with dummy data
12941324
target = Target(
12951325
head='e279a7e06e61c14868ca7d71dea795420aea6539',
1296-
branch='master',
1326+
branch=repo.default_branch_name,
12971327
remote='https://github.com/apache/arrow',
12981328
version='1.0.0dev123',
12991329
r_version='0.13.0.100000123',

‎dev/archery/archery/docker/cli.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -217,7 +217,8 @@ def docker_run(obj, image, command, *, env, user, force_pull, force_build,
217217
PYTHON=3.8 archery docker run conda-python
218218
219219
# disable the cache only for the leaf image
220-
PANDAS=master archery docker run --no-leaf-cache conda-python-pandas
220+
PANDAS=upstream_devel archery docker run --no-leaf-cache \
221+
conda-python-pandas
221222
222223
# entirely skip building the image
223224
archery docker run --no-pull --no-build conda-python

‎dev/archery/archery/docker/tests/test_docker.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -259,12 +259,12 @@ def test_arrow_example_validation_passes(arrow_compose_path):
259259
def test_compose_default_params_and_env(arrow_compose_path):
260260
compose = DockerCompose(arrow_compose_path, params=dict(
261261
UBUNTU='18.04',
262-
DASK='master'
262+
DASK='upstream_devel'
263263
))
264264
assert compose.config.dotenv == arrow_compose_env
265265
assert compose.config.params == {
266266
'UBUNTU': '18.04',
267-
'DASK': 'master',
267+
'DASK': 'upstream_devel',
268268
}
269269

270270

@@ -492,7 +492,7 @@ def test_compose_push(arrow_compose_path):
492492
def test_compose_error(arrow_compose_path):
493493
compose = DockerCompose(arrow_compose_path, params=dict(
494494
PYTHON='3.8',
495-
PANDAS='master'
495+
PANDAS='upstream_devel'
496496
))
497497

498498
error = subprocess.CalledProcessError(99, [])
@@ -503,7 +503,7 @@ def test_compose_error(arrow_compose_path):
503503
exception_message = str(exc.value)
504504
assert "exited with a non-zero exit code 99" in exception_message
505505
assert "PANDAS: latest" in exception_message
506-
assert "export PANDAS=master" in exception_message
506+
assert "export PANDAS=upstream_devel" in exception_message
507507

508508

509509
def test_image_with_gpu(arrow_compose_path):

0 commit comments

Comments
 (0)