Skip to content

[backport-security_detection_engine-9.4] Sync CI configuration with main branch - #20416

Merged
mrodm merged 5 commits into
elastic:backport-security_detection_engine-9.4from
mrodm:sync-ci-configuration-security_detection_engine-9.4
Sep 8, 2026
Merged

mrodm merged 5 commits into
elastic:backport-security_detection_engine-9.4from
mrodm:sync-ci-configuration-security_detection_engine-9.4

Conversation

@mrodm

@mrodm mrodm commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Update .buildkite and .github folders with the latest CI configuration
from the main branch to the backport-security_detection_engine-9.4 backport branch.

Proposed commit message

Sync CI configuration with main branch for backport-security_detection_engine-9.4.

Keep CI pipelines, GitHub Actions, and tooling consistent between active backport
branches and main, so that automated backport workflows and tooling improvements
land on all supported branches.

Author's Checklist

  • CI configuration files have been synced verbatim from main.

Related issues


This PR was generated with the assistance of Claude (claude-sonnet-4-6).

@mrodm mrodm self-assigned this Jul 30, 2026
@mrodm
mrodm marked this pull request as ready for review July 31, 2026 09:45
@mrodm
mrodm requested a review from a team as a code owner July 31, 2026 09:45
@mrodm
mrodm marked this pull request as draft August 5, 2026 16:23
@mrodm
mrodm force-pushed the sync-ci-configuration-security_detection_engine-9.4 branch from a214753 to 4df2a94 Compare August 13, 2026 14:25
@mrodm
mrodm force-pushed the sync-ci-configuration-security_detection_engine-9.4 branch from 4df2a94 to 6f1289d Compare September 4, 2026 12:45
@mrodm

mrodm commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

/test

@mrodm
mrodm marked this pull request as ready for review September 4, 2026 15:45
Copilot AI lite review requested due to automatic review settings September 4, 2026 15:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The backport-branch sync script’s whitespace-matching regexes use \s with grep -E/sed, which won’t match as intended and will break the version-pin preservation logic.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Syncs CI and backport-automation configuration from main onto the backport-security_detection_engine-9.4 branch, including migrating automation logic into a standalone cmd/backport sub-module so backport branches can run the same workflows and tooling as main.

Changes:

  • Adds/updates GitHub Actions workflows and a composite action to support backport changelog syncing (including retry via /sync-changelog) and automated backport PR creation.
  • Introduces a self-contained cmd/backport Go sub-module and updates Buildkite/GitHub workflows/scripts to use the backport binary instead of mage targets.
  • Updates Buildkite pipelines/scripts to optimize package selection and to add new checks around changelog and backport tooling.
File summaries
File Description
go.sum Drops root-module deps that moved into cmd/backport sub-module.
go.mod Removes go-gh/safeexec from root module requirements.
dev/testsreporter/xunit.go Removed (functionality moved out of root toolchain).
dev/testsreporter/testsreporter.go Removed (functionality moved out of root toolchain).
dev/testsreporter/testdata/sql_system_1718901944954979316.xml Removed testsreporter fixture data.
dev/testsreporter/testdata/nats_1718676056733968706.xml Removed testsreporter fixture data.
dev/testsreporter/testdata/fortinet_fortigate_1718675058322143368.xml Removed testsreporter fixture data.
dev/testsreporter/testdata/fortinet_fortigate_1718675052813006253.xml Removed testsreporter fixture data.
dev/testsreporter/testdata/elastic_package_registry_1718676502022628094.xml Removed testsreporter fixture data.
dev/testsreporter/testdata/CODEOWNERS-default-tests Removed testsreporter CODEOWNERS fixture.
dev/testsreporter/testdata/cisco_umbrella_1718676502022628093.xml Removed testsreporter fixture data.
dev/testsreporter/testdata/cisco_umbrella_1718675095255991383.xml Removed testsreporter fixture data.
dev/testsreporter/testdata/cisco_umbrella_1718675095080627646.xml Removed testsreporter fixture data.
dev/testsreporter/testdata/cisco_umbrella_1718675092421310216.xml Removed testsreporter fixture data.
dev/testsreporter/reporter.go Removed testsreporter implementation.
dev/testsreporter/reporter_test.go Removed testsreporter unit tests.
dev/testsreporter/packageerror.go Removed testsreporter implementation.
dev/testsreporter/packageerror_test.go Removed testsreporter unit tests.
dev/testsreporter/githubissue.go Removed testsreporter implementation.
dev/testsreporter/github.go Removed testsreporter GH client wrapper.
dev/testsreporter/github_test.go Removed testsreporter GH wrapper tests.
dev/testsreporter/format.go Removed testsreporter templating code.
dev/testsreporter/errorlinks.go Removed testsreporter link tracking.
dev/testsreporter/dataerror.go Removed testsreporter shared error data.
dev/testsreporter/builderror.go Removed testsreporter build error wrapper.
dev/testsreporter/builderror_test.go Removed testsreporter unit tests.
dev/testsreporter/_static/summary.tmpl Removed testsreporter template.
dev/testsreporter/_static/description.tmpl Removed testsreporter template.
dev/scripts/backport_apply.sh Switches wrapper from mage ApplyBackport to backport apply and builds binary if needed.
dev/requiresupdate/testdata/CODEOWNERS Removed requiresupdate fixture.
dev/requiresupdate/requiresupdate.go Removed requiresupdate implementation from root tooling.
dev/requiresupdate/requiresupdate_test.go Removed requiresupdate unit tests.
dev/requiresupdate/publish_test.go Removed requiresupdate publishing tests.
dev/packagenames/packagenames.go Replaces slices.Sort with sort.Strings for broader Go compatibility.
dev/codeowners/codeowners.go Removes exported ParseOwners from root dev package.
dev/codeowners/codeowners_test.go Refactors tests to use an internal helper instead of removed API.
dev/citools/packages.go Replaces slices.Sort with sort.Strings.
dev/citools/packagemanifest.go Makes manifest parse helper unexported and updates call site.
dev/citools/gomod_test.go Makes test fixture’s go directive compatible across Go toolchains.
dev/backports/checklist/checklist_test.go Removed legacy backports checklist tests (migrated to cmd/backport).
dev/backports/changelog/collect_test.go Removed legacy backports changelog tests (migrated to cmd/backport).
cmd/backport/tools.go Fixes file purpose: uses tools build tag and tracks tool deps for sub-module.
cmd/backport/magefile.go Adds mage targets for cmd/backport sub-module dev workflow.
cmd/backport/go.mod Adds standalone Go module for backport tooling and its dependencies.
cmd/backport/gitutil/git.go Adds RunToStderr and documents local-copy intent.
cmd/backport/codeowners/codeowners.go Adds local CODEOWNERS parser copy for standalone tool.
cmd/backport/citools/packages.go Adds local package-discovery copy for standalone tool.
cmd/backport/citools/packagemanifest.go Adds local manifest parser copy for standalone tool.
cmd/backport/backports/packages/detect.go Switches to cmd/backport’s local citools package.
cmd/backport/backports/packages/detect_test.go Adds unit tests for package detection logic.
cmd/backport/backports/owners/owners.go Switches to cmd/backport’s local citools/codeowners.
cmd/backport/backports/owners/owners_test.go Switches tests to local cmd/backport/codeowners.
cmd/backport/backports/owners/compare.go Switches to local codeowners/gitutil.
cmd/backport/backports/owners/compare_integration_test.go Switches to local gitutil.
cmd/backport/backports/owners/check.go Switches to local codeowners/gitutil.
cmd/backport/backports/owners/check_test.go Adds unit tests for team de-duplication behavior.
cmd/backport/backports/inventory.go Adds skip_checklist_packages support and exposes it via helper.
cmd/backport/backports/inventory_test.go Adds tests for new ListSkipChecklistPackages behavior.
cmd/backport/backports/checklist/checklist.go Restores checkbox rendering + adds parsing/status update helpers.
cmd/backport/backports/changelog/update.go Adds helper to rewrite link: fields within a specific version entry.
cmd/backport/backports/changelog/update_test.go Adds tests for changelog link updating.
cmd/backport/backports/changelog/sync.go Uses RunToStderr and force-push semantics for machine-owned branches.
cmd/backport/backports/changelog/sync_test.go Adds unit tests for TSV parsing and title/commit message formatting.
cmd/backport/backports/changelog/resolve.go Switches to local citools for package indexing.
cmd/backport/backports/changelog/resolve_test.go Adds tests for package index behavior (nested packages, name mismatch).
cmd/backport/backports/changelog/insert.go Adds semver-aware changelog entry insertion helper.
cmd/backport/backports/changelog/insert_test.go Adds tests for insertion ordering/idempotency.
cmd/backport/backports/changelog/extract.go Adds diff parser to extract newly-added changelog entries.
cmd/backport/backports/changelog/extract_test.go Adds tests for changelog diff extraction.
cmd/backport/backports/changelog/comment.go Extends comment messaging for “already exists” and retry hint.
cmd/backport/backports/changelog/comment_test.go Updates tests for new comment outcomes and retry hint.
cmd/backport/backports/changelog/collect.go Adds retry/backoff for PR lookup, avoids duplicates, and uses ... diff.
cmd/backport/backports/changelog/collect_test.go Adds tests for checking versions already present on main.
cmd/backport/backports/apply/apply_test.go Adds sentinel URL test and switches imports to new owners package path.
cmd/backport/.go-version Pins Go toolchain version for the standalone backport tool.
.gitignore Ignores .claude/ state but keeps /.claude/settings.json tracked.
.github/workflows/sync-backport-changelog.yml Adds composite-action usage + adds /sync-changelog retry flow.
.github/workflows/requires-update.yml Re-enables scheduled runs with weekday cron.
.github/workflows/post-backport-checklist.yml Builds cmd/backport and re-creates checklist comments using the binary.
.github/workflows/bump-elastic-stack-version.yml Updates pinned updatecli action SHA/version.
.github/workflows/backport-packages-detect.yml Builds cmd/backport and uses it for package detection output.
.github/workflows/auto-backport.yml Adds workflow to auto-create backport PRs based on checklist edits / main merges.
.github/actions/sync-backport-changelog/action.yml Adds composite action wrapping tool build, sync, and commenting.
.buildkite/scripts/trigger_integrations_in_parallel.sh Computes affected packages from merge-base diff to reduce workload.
.buildkite/scripts/trigger_backport.sh Switches to with_backport and restricts branch creation to dispatch pipeline.
.buildkite/scripts/trigger_backport_lib.sh Switches inventory validation/active checks from mage to backport CLI.
.buildkite/scripts/test_trigger_backport.sh Updates mocks from mage to backport subcommands.
.buildkite/scripts/test_non_package_patterns.sh Updates non-package path fixtures for new backport/tooling paths.
.buildkite/scripts/test_integrations_with_serverless.sh Computes affected packages from merge-base diff to reduce workload.
.buildkite/scripts/packages/crowdstrike.sh Adds package-specific CI check to ensure generated pipelines stay in sync.
.buildkite/scripts/non_package_patterns.txt Expands “non-package” patterns for new backport tooling and CI paths.
.buildkite/scripts/common.sh Adds with_backport, exports BACKPORT_BIN, and reuses COMMIT_MERGE.
.buildkite/scripts/check_changelog_versions_in_main.sh Adds Buildkite step script to prevent duplicate changelog versions vs main.
.buildkite/scripts/check_changelog_entries.sh Adds hard-fail behavior for REPLACE_ME sentinel links and exit-code split.
.buildkite/scripts/check_backports_inventory.sh Switches inventory validation from mage to backport validate-inventory.
.buildkite/scripts/check_backport_tool.sh Adds a Buildkite check step for cmd/backport sub-module (mage check + clean tree).
.buildkite/scripts/check_backport_owners.sh Switches owners check execution from mage to backport check-owners.
.buildkite/scripts/backport_branch.sh Syncs .buildkite, .github, and cmd/backport; removes dev backport tooling on branch.
.buildkite/pull-requests.json Updates skip patterns and adds disabled pipeline entry for backport-dispatch.
.buildkite/pipeline.yml Adds backport tool tests + adjusts inventory validation paths + changelog checks.
.buildkite/pipeline.serverless.yml Updates pinned KIND/K8S versions for serverless pipeline.
.buildkite/pipeline.schedule-weekly.yml Updates scheduled stack snapshot version.
.buildkite/pipeline.schedule-daily.yml Updates scheduled stack snapshot version.
.buildkite/pipeline.backport.yml Adds notifications + allows triggers from new dispatch pipeline.
.buildkite/pipeline.backport-dispatch.yml Adds a dedicated pipeline for automated backport branch creation.
Review details

Suppressed comments (2)

.buildkite/scripts/backport_branch.sh:170

  • Same issue here: sed regular expressions don’t treat \s as whitespace, so the K8S_VERSION line replacement won’t match and the preserved pin won’t be restored.
    echo "--- Restoring K8S_VERSION in ${pipeline_yml}..."
    sed -i "s|^\s*K8S_VERSION:.*|${k8s_version_line}|" "${pipeline_yml}"
  fi

.buildkite/scripts/backport_branch.sh:174

  • Same sed regex issue for KIND_VERSION: \s won’t match whitespace, so this restore step will be a no-op.
  if [ -n "${kind_version_line}" ]; then
    echo "--- Restoring KIND_VERSION in ${pipeline_yml}..."
    sed -i "s|^\s*KIND_VERSION:.*|${kind_version_line}|" "${pipeline_yml}"
  fi
  • Files reviewed: 103/114 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +16 to +20
if_changed:
- ".backports.yml"
- ".buildkite/scripts/run_dev_scripts_tests.sh"
- "dev/scripts/**.sh"
- "dev/backports/**"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened PR for this #21078
No need to update it here.

Comment on lines +157 to +158
k8s_version_line=$(grep -E '^\s*K8S_VERSION:' "${pipeline_yml}" || true)
kind_version_line=$(grep -E '^\s*KIND_VERSION:' "${pipeline_yml}" || true)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works:

 $ grep -E '^\s*K8S_VERSION:' ".buildkite/pipeline.yml" || true  
  K8S_VERSION: 'v1.36.1'

Copilot AI review requested due to automatic review settings September 7, 2026 14:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new package-selection logic in Buildkite scripts uses pr_has_package_related_files with inverted semantics, which can unintentionally skip integration testing for some diffs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 106/116 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment on lines +84 to +86
if [[ "${FORCE_CHECK_ALL}" == "true" ]] || echo "${changed_files}" | pr_has_package_related_files; then
echo "Non-package files changed or FORCE_CHECK_ALL set: scanning all packages"
PACKAGE_LIST=$(list_all_directories)
Comment on lines +53 to +55
if [[ "${FORCE_CHECK_ALL}" == "true" ]] || echo "${changed_files}" | pr_has_package_related_files; then
echo "Non-package files changed or FORCE_CHECK_ALL set: scanning all packages"
PACKAGE_LIST=$(list_all_directories)
Copilot AI review requested due to automatic review settings September 7, 2026 14:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

.github/workflows/requires-update.yml invokes a mage RequiresUpdate target that does not exist on this branch (and dev/requiresupdate/ was removed), so the workflow cannot run successfully from this ref.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 106/116 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment on lines 57 to 64
- name: Run RequiresUpdate
# Scans packages, applies proposals, and opens one PR (or issue) per
# package — see dev/requiresupdate.
# Scheduled runs (when cron is enabled) always use dryRun=false and
# preview=false; workflow_dispatch inputs are only read for manual runs.
# Scheduled runs always use dryRun=false and preview=false;
# workflow_dispatch inputs are only read for manual runs.
id: requires_update
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expected backport branches do not need to to run this

Copilot AI review requested due to automatic review settings September 8, 2026 10:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The changes consistently align backport-branch CI/backport automation with main, and the updated workflows/scripts/tooling appear internally consistent and covered by added/updated tests.

Review details
  • Files reviewed: 105/115 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

💚 Build Succeeded

History

cc @mrodm

@mergify

mergify Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@mrodm
mrodm merged commit 0211546 into elastic:backport-security_detection_engine-9.4 Sep 8, 2026
9 checks passed
@mrodm
mrodm deleted the sync-ci-configuration-security_detection_engine-9.4 branch September 8, 2026 14:22
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Changelog sync skipped — all changelog versions are already present on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants