Repository navigation
[backport-security_detection_engine-9.4] Sync CI configuration with main branch - #20416
Conversation
a214753 to
4df2a94
Compare
4df2a94 to
6f1289d
Compare
|
/test |
There was a problem hiding this comment.
🟡 Changes recommended
The backport-branch sync script’s whitespace-matching regexes use \s with grep -E/sed, which won’t match as intended and will break the version-pin preservation logic.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Syncs CI and backport-automation configuration from main onto the backport-security_detection_engine-9.4 branch, including migrating automation logic into a standalone cmd/backport sub-module so backport branches can run the same workflows and tooling as main.
Changes:
- Adds/updates GitHub Actions workflows and a composite action to support backport changelog syncing (including retry via
/sync-changelog) and automated backport PR creation. - Introduces a self-contained
cmd/backportGo sub-module and updates Buildkite/GitHub workflows/scripts to use thebackportbinary instead ofmagetargets. - Updates Buildkite pipelines/scripts to optimize package selection and to add new checks around changelog and backport tooling.
File summaries
| File | Description |
|---|---|
| go.sum | Drops root-module deps that moved into cmd/backport sub-module. |
| go.mod | Removes go-gh/safeexec from root module requirements. |
| dev/testsreporter/xunit.go | Removed (functionality moved out of root toolchain). |
| dev/testsreporter/testsreporter.go | Removed (functionality moved out of root toolchain). |
| dev/testsreporter/testdata/sql_system_1718901944954979316.xml | Removed testsreporter fixture data. |
| dev/testsreporter/testdata/nats_1718676056733968706.xml | Removed testsreporter fixture data. |
| dev/testsreporter/testdata/fortinet_fortigate_1718675058322143368.xml | Removed testsreporter fixture data. |
| dev/testsreporter/testdata/fortinet_fortigate_1718675052813006253.xml | Removed testsreporter fixture data. |
| dev/testsreporter/testdata/elastic_package_registry_1718676502022628094.xml | Removed testsreporter fixture data. |
| dev/testsreporter/testdata/CODEOWNERS-default-tests | Removed testsreporter CODEOWNERS fixture. |
| dev/testsreporter/testdata/cisco_umbrella_1718676502022628093.xml | Removed testsreporter fixture data. |
| dev/testsreporter/testdata/cisco_umbrella_1718675095255991383.xml | Removed testsreporter fixture data. |
| dev/testsreporter/testdata/cisco_umbrella_1718675095080627646.xml | Removed testsreporter fixture data. |
| dev/testsreporter/testdata/cisco_umbrella_1718675092421310216.xml | Removed testsreporter fixture data. |
| dev/testsreporter/reporter.go | Removed testsreporter implementation. |
| dev/testsreporter/reporter_test.go | Removed testsreporter unit tests. |
| dev/testsreporter/packageerror.go | Removed testsreporter implementation. |
| dev/testsreporter/packageerror_test.go | Removed testsreporter unit tests. |
| dev/testsreporter/githubissue.go | Removed testsreporter implementation. |
| dev/testsreporter/github.go | Removed testsreporter GH client wrapper. |
| dev/testsreporter/github_test.go | Removed testsreporter GH wrapper tests. |
| dev/testsreporter/format.go | Removed testsreporter templating code. |
| dev/testsreporter/errorlinks.go | Removed testsreporter link tracking. |
| dev/testsreporter/dataerror.go | Removed testsreporter shared error data. |
| dev/testsreporter/builderror.go | Removed testsreporter build error wrapper. |
| dev/testsreporter/builderror_test.go | Removed testsreporter unit tests. |
| dev/testsreporter/_static/summary.tmpl | Removed testsreporter template. |
| dev/testsreporter/_static/description.tmpl | Removed testsreporter template. |
| dev/scripts/backport_apply.sh | Switches wrapper from mage ApplyBackport to backport apply and builds binary if needed. |
| dev/requiresupdate/testdata/CODEOWNERS | Removed requiresupdate fixture. |
| dev/requiresupdate/requiresupdate.go | Removed requiresupdate implementation from root tooling. |
| dev/requiresupdate/requiresupdate_test.go | Removed requiresupdate unit tests. |
| dev/requiresupdate/publish_test.go | Removed requiresupdate publishing tests. |
| dev/packagenames/packagenames.go | Replaces slices.Sort with sort.Strings for broader Go compatibility. |
| dev/codeowners/codeowners.go | Removes exported ParseOwners from root dev package. |
| dev/codeowners/codeowners_test.go | Refactors tests to use an internal helper instead of removed API. |
| dev/citools/packages.go | Replaces slices.Sort with sort.Strings. |
| dev/citools/packagemanifest.go | Makes manifest parse helper unexported and updates call site. |
| dev/citools/gomod_test.go | Makes test fixture’s go directive compatible across Go toolchains. |
| dev/backports/checklist/checklist_test.go | Removed legacy backports checklist tests (migrated to cmd/backport). |
| dev/backports/changelog/collect_test.go | Removed legacy backports changelog tests (migrated to cmd/backport). |
| cmd/backport/tools.go | Fixes file purpose: uses tools build tag and tracks tool deps for sub-module. |
| cmd/backport/magefile.go | Adds mage targets for cmd/backport sub-module dev workflow. |
| cmd/backport/go.mod | Adds standalone Go module for backport tooling and its dependencies. |
| cmd/backport/gitutil/git.go | Adds RunToStderr and documents local-copy intent. |
| cmd/backport/codeowners/codeowners.go | Adds local CODEOWNERS parser copy for standalone tool. |
| cmd/backport/citools/packages.go | Adds local package-discovery copy for standalone tool. |
| cmd/backport/citools/packagemanifest.go | Adds local manifest parser copy for standalone tool. |
| cmd/backport/backports/packages/detect.go | Switches to cmd/backport’s local citools package. |
| cmd/backport/backports/packages/detect_test.go | Adds unit tests for package detection logic. |
| cmd/backport/backports/owners/owners.go | Switches to cmd/backport’s local citools/codeowners. |
| cmd/backport/backports/owners/owners_test.go | Switches tests to local cmd/backport/codeowners. |
| cmd/backport/backports/owners/compare.go | Switches to local codeowners/gitutil. |
| cmd/backport/backports/owners/compare_integration_test.go | Switches to local gitutil. |
| cmd/backport/backports/owners/check.go | Switches to local codeowners/gitutil. |
| cmd/backport/backports/owners/check_test.go | Adds unit tests for team de-duplication behavior. |
| cmd/backport/backports/inventory.go | Adds skip_checklist_packages support and exposes it via helper. |
| cmd/backport/backports/inventory_test.go | Adds tests for new ListSkipChecklistPackages behavior. |
| cmd/backport/backports/checklist/checklist.go | Restores checkbox rendering + adds parsing/status update helpers. |
| cmd/backport/backports/changelog/update.go | Adds helper to rewrite link: fields within a specific version entry. |
| cmd/backport/backports/changelog/update_test.go | Adds tests for changelog link updating. |
| cmd/backport/backports/changelog/sync.go | Uses RunToStderr and force-push semantics for machine-owned branches. |
| cmd/backport/backports/changelog/sync_test.go | Adds unit tests for TSV parsing and title/commit message formatting. |
| cmd/backport/backports/changelog/resolve.go | Switches to local citools for package indexing. |
| cmd/backport/backports/changelog/resolve_test.go | Adds tests for package index behavior (nested packages, name mismatch). |
| cmd/backport/backports/changelog/insert.go | Adds semver-aware changelog entry insertion helper. |
| cmd/backport/backports/changelog/insert_test.go | Adds tests for insertion ordering/idempotency. |
| cmd/backport/backports/changelog/extract.go | Adds diff parser to extract newly-added changelog entries. |
| cmd/backport/backports/changelog/extract_test.go | Adds tests for changelog diff extraction. |
| cmd/backport/backports/changelog/comment.go | Extends comment messaging for “already exists” and retry hint. |
| cmd/backport/backports/changelog/comment_test.go | Updates tests for new comment outcomes and retry hint. |
| cmd/backport/backports/changelog/collect.go | Adds retry/backoff for PR lookup, avoids duplicates, and uses ... diff. |
| cmd/backport/backports/changelog/collect_test.go | Adds tests for checking versions already present on main. |
| cmd/backport/backports/apply/apply_test.go | Adds sentinel URL test and switches imports to new owners package path. |
| cmd/backport/.go-version | Pins Go toolchain version for the standalone backport tool. |
| .gitignore | Ignores .claude/ state but keeps /.claude/settings.json tracked. |
| .github/workflows/sync-backport-changelog.yml | Adds composite-action usage + adds /sync-changelog retry flow. |
| .github/workflows/requires-update.yml | Re-enables scheduled runs with weekday cron. |
| .github/workflows/post-backport-checklist.yml | Builds cmd/backport and re-creates checklist comments using the binary. |
| .github/workflows/bump-elastic-stack-version.yml | Updates pinned updatecli action SHA/version. |
| .github/workflows/backport-packages-detect.yml | Builds cmd/backport and uses it for package detection output. |
| .github/workflows/auto-backport.yml | Adds workflow to auto-create backport PRs based on checklist edits / main merges. |
| .github/actions/sync-backport-changelog/action.yml | Adds composite action wrapping tool build, sync, and commenting. |
| .buildkite/scripts/trigger_integrations_in_parallel.sh | Computes affected packages from merge-base diff to reduce workload. |
| .buildkite/scripts/trigger_backport.sh | Switches to with_backport and restricts branch creation to dispatch pipeline. |
| .buildkite/scripts/trigger_backport_lib.sh | Switches inventory validation/active checks from mage to backport CLI. |
| .buildkite/scripts/test_trigger_backport.sh | Updates mocks from mage to backport subcommands. |
| .buildkite/scripts/test_non_package_patterns.sh | Updates non-package path fixtures for new backport/tooling paths. |
| .buildkite/scripts/test_integrations_with_serverless.sh | Computes affected packages from merge-base diff to reduce workload. |
| .buildkite/scripts/packages/crowdstrike.sh | Adds package-specific CI check to ensure generated pipelines stay in sync. |
| .buildkite/scripts/non_package_patterns.txt | Expands “non-package” patterns for new backport tooling and CI paths. |
| .buildkite/scripts/common.sh | Adds with_backport, exports BACKPORT_BIN, and reuses COMMIT_MERGE. |
| .buildkite/scripts/check_changelog_versions_in_main.sh | Adds Buildkite step script to prevent duplicate changelog versions vs main. |
| .buildkite/scripts/check_changelog_entries.sh | Adds hard-fail behavior for REPLACE_ME sentinel links and exit-code split. |
| .buildkite/scripts/check_backports_inventory.sh | Switches inventory validation from mage to backport validate-inventory. |
| .buildkite/scripts/check_backport_tool.sh | Adds a Buildkite check step for cmd/backport sub-module (mage check + clean tree). |
| .buildkite/scripts/check_backport_owners.sh | Switches owners check execution from mage to backport check-owners. |
| .buildkite/scripts/backport_branch.sh | Syncs .buildkite, .github, and cmd/backport; removes dev backport tooling on branch. |
| .buildkite/pull-requests.json | Updates skip patterns and adds disabled pipeline entry for backport-dispatch. |
| .buildkite/pipeline.yml | Adds backport tool tests + adjusts inventory validation paths + changelog checks. |
| .buildkite/pipeline.serverless.yml | Updates pinned KIND/K8S versions for serverless pipeline. |
| .buildkite/pipeline.schedule-weekly.yml | Updates scheduled stack snapshot version. |
| .buildkite/pipeline.schedule-daily.yml | Updates scheduled stack snapshot version. |
| .buildkite/pipeline.backport.yml | Adds notifications + allows triggers from new dispatch pipeline. |
| .buildkite/pipeline.backport-dispatch.yml | Adds a dedicated pipeline for automated backport branch creation. |
Review details
Suppressed comments (2)
.buildkite/scripts/backport_branch.sh:170
- Same issue here:
sedregular expressions don’t treat\sas whitespace, so the K8S_VERSION line replacement won’t match and the preserved pin won’t be restored.
echo "--- Restoring K8S_VERSION in ${pipeline_yml}..."
sed -i "s|^\s*K8S_VERSION:.*|${k8s_version_line}|" "${pipeline_yml}"
fi
.buildkite/scripts/backport_branch.sh:174
- Same
sedregex issue for KIND_VERSION:\swon’t match whitespace, so this restore step will be a no-op.
if [ -n "${kind_version_line}" ]; then
echo "--- Restoring KIND_VERSION in ${pipeline_yml}..."
sed -i "s|^\s*KIND_VERSION:.*|${kind_version_line}|" "${pipeline_yml}"
fi
- Files reviewed: 103/114 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if_changed: | ||
| - ".backports.yml" | ||
| - ".buildkite/scripts/run_dev_scripts_tests.sh" | ||
| - "dev/scripts/**.sh" | ||
| - "dev/backports/**" |
There was a problem hiding this comment.
Opened PR for this #21078
No need to update it here.
| k8s_version_line=$(grep -E '^\s*K8S_VERSION:' "${pipeline_yml}" || true) | ||
| kind_version_line=$(grep -E '^\s*KIND_VERSION:' "${pipeline_yml}" || true) |
There was a problem hiding this comment.
It works:
$ grep -E '^\s*K8S_VERSION:' ".buildkite/pipeline.yml" || true
K8S_VERSION: 'v1.36.1'
There was a problem hiding this comment.
🟡 Changes recommended
The new package-selection logic in Buildkite scripts uses pr_has_package_related_files with inverted semantics, which can unintentionally skip integration testing for some diffs.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 106/116 changed files
- Comments generated: 2
- Review effort level: Lite
| if [[ "${FORCE_CHECK_ALL}" == "true" ]] || echo "${changed_files}" | pr_has_package_related_files; then | ||
| echo "Non-package files changed or FORCE_CHECK_ALL set: scanning all packages" | ||
| PACKAGE_LIST=$(list_all_directories) |
| if [[ "${FORCE_CHECK_ALL}" == "true" ]] || echo "${changed_files}" | pr_has_package_related_files; then | ||
| echo "Non-package files changed or FORCE_CHECK_ALL set: scanning all packages" | ||
| PACKAGE_LIST=$(list_all_directories) |
There was a problem hiding this comment.
🟡 Changes recommended
.github/workflows/requires-update.yml invokes a mage RequiresUpdate target that does not exist on this branch (and dev/requiresupdate/ was removed), so the workflow cannot run successfully from this ref.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 106/116 changed files
- Comments generated: 1
- Review effort level: Lite
| - name: Run RequiresUpdate | ||
| # Scans packages, applies proposals, and opens one PR (or issue) per | ||
| # package — see dev/requiresupdate. | ||
| # Scheduled runs (when cron is enabled) always use dryRun=false and | ||
| # preview=false; workflow_dispatch inputs are only read for manual runs. | ||
| # Scheduled runs always use dryRun=false and preview=false; | ||
| # workflow_dispatch inputs are only read for manual runs. | ||
| id: requires_update | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} |
There was a problem hiding this comment.
Expected backport branches do not need to to run this
There was a problem hiding this comment.
🟢 Approval recommended
The changes consistently align backport-branch CI/backport automation with main, and the updated workflows/scripts/tooling appear internally consistent and covered by added/updated tests.
Review details
- Files reviewed: 105/115 changed files
- Comments generated: 0 new
- Review effort level: Lite
💚 Build Succeeded
History
cc @mrodm |
|
Tick the box to add this pull request to the merge queue (same as
|
0211546
into
elastic:backport-security_detection_engine-9.4
|
Changelog sync skipped — all changelog versions are already present on |
Update
.buildkiteand.githubfolders with the latest CI configurationfrom the
mainbranch to thebackport-security_detection_engine-9.4backport branch.Proposed commit message
Sync CI configuration with main branch for backport-security_detection_engine-9.4.
Keep CI pipelines, GitHub Actions, and tooling consistent between active backport
branches and
main, so that automated backport workflows and tooling improvementsland on all supported branches.
Author's Checklist
main.Related issues