Skip to content

add processor tags and preserve_original_event on failure (1/4) - #20572

Merged
kcreddy merged 8 commits into
elastic:mainfrom
kcreddy:tagging-evt-original-standards
Aug 20, 2026
Merged

kcreddy merged 8 commits into
elastic:mainfrom
kcreddy:tagging-evt-original-standards

Conversation

@kcreddy

@kcreddy kcreddy commented Aug 6, 2026 •

Copy link
Copy Markdown
Contributor

Proposed commit message

ssi: add processor tags and preserve_original_event on failure (1/4)

Tag every ingest pipeline processor across 51 SSI-owned packages
(1password through entityanalytics_entra_id), so that failure telemetry
can attribute an error to the step that produced it rather than
collapsing same-type processors into one bucket. All 25752 processors
across the 303 pipeline files now carry a tag, 10853 of them newly,
including those nested inside `on_failure` handlers and `foreach`
bodies, and those in the pipeline-level `on_failure` block.

Every processor gets a tag of the form `<descriptive_name>_<8hex>`,
where the 8-hex suffix is a content hash over the processor body and
its enclosing context. Identical constructs in identical surroundings
produce the same hash in every package, giving each tag a stable global
identity in failure telemetry. Tags that already carried a hash are
left byte-identical -- 3275 of them, in aws, crowdstrike and axonius,
which shipped tags before this series; a hash is only meaningful
relative to the generator that produced it, so re-hashing a published
tag churns a value consumers may key off. Those three packages therefore
keep the earlier suffix format. Tags that would have said nothing
(`script_<hash>`, `fail_<hash>`) are seeded from the processor's
description, name, message or preceding comment, and tags whose leading
word named the wrong action (`set_...` on an append) are corrected.

Add preserve_original_event to pipeline-level on_failure handlers that
were missing it, ensuring the raw payload is retained when a pipeline
error document is indexed. Sixteen of the 51 packages needed it.

Six processors across five packages were exact duplicates of an earlier
sibling and could never have any effect; they are removed rather than
given a disambiguating tag. Five further fixes ride along, each in the
affected package's changelog:

  - aws, aws_bedrock: five pipeline-level on_failure handlers set
    error.message instead of appending, discarding what processor-level
    handlers recorded. error.message is now an array on failed
    documents.
  - azure: the aadgraphactivitylogs link to the shared pipeline is
    refreshed, since tagging that pipeline changed its checksum.
  - carbon_black_cloud/watchlist_hit: the append_related_hosts processor
    was gated on ctx.host?.hostname but appended user.domain; guard
    corrected to ctx.user?.domain, matching alert_v7 and endpoint_event.
  - azure/signinlogs: tag azure-json-keys-to-snake-case renamed to
    azure_json_keys_to_snake_case (hyphens are not valid in tag names).
  - claroty_ctd/event: tag script_to_drop_N/A_values_fields renamed to
    script_to_drop_NA_values_fields (slash is not a valid identifier
    character).

These are enhancements, so the packages take a minor version bump.

Updates #20558

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Tag every ingest pipeline processor across 51 SSI-owned packages
(1password through entityanalytics_entra_id) with a unique,
descriptive identifier so that failure telemetry can attribute
errors to the specific step that failed rather than collapsing
same-type processors into one bucket.

Add preserve_original_event to pipeline-level on_failure handlers
that were missing it, ensuring the raw payload is retained when a
pipeline error document is indexed.

Updates elastic#20558
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!


The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

kcreddy added 3 commits August 6, 2026 19:44
Update the link field in each new changelog entry from the tracking
issue to the merged pull request.

Updates elastic#20558
Tag every ingest pipeline processor across 51 SSI-owned packages
(1password through entityanalytics_entra_id), so that failure telemetry
can attribute an error to the step that produced it rather than
collapsing same-type processors into one bucket. All 25752 processors
across the 303 pipeline files now carry a tag, 10853 of them newly,
including those nested inside `on_failure` handlers and `foreach`
bodies, and those in the pipeline-level `on_failure` block.

Existing tags keep their name. Where a name was not already unique in
its pipeline, or the processor had no tag at all, it gains an 8-hex
suffix hashed over the processor's content and everything enclosing it.
Tags that already carried a hash are left byte-identical -- 3275 of
them, in aws, crowdstrike and axonius, which shipped tags before this
series; a hash is only meaningful relative to the generator that
produced it, so re-hashing a published tag churns a value consumers may
key off. Those three packages therefore keep the earlier suffix format.
Tags that would have said nothing (`script_<hash>`, `fail_<hash>`) are
seeded from the processor's description, name, message or preceding
comment, and tags whose leading word named the wrong action (`set_...`
on an append) are corrected.

Add preserve_original_event to pipeline-level on_failure handlers that
were missing it, ensuring the raw payload is retained when a pipeline
error document is indexed. Sixteen of the 51 packages needed it.

Six processors across five packages were exact duplicates of an earlier
sibling and could never have any effect; they are removed rather than
given a disambiguating tag. Two further fixes ride along, each in the
affected package's changelog:

  - aws, aws_bedrock: five pipeline-level on_failure handlers set
    error.message instead of appending, discarding what processor-level
    handlers recorded. error.message is now an array on failed
    documents.
  - azure: the aadgraphactivitylogs link to the shared pipeline is
    refreshed, since tagging that pipeline changed its checksum.

These are enhancements, so the packages take a minor version bump.

Updates elastic#20558
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Aug 13, 2026 •

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

Package entityanalytics_entra_id 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
entity 8928.57 7092.2 -1836.37 (-20.57%) 💔

To see the full report comment with /test benchmark fullreport

@kcreddy kcreddy self-assigned this Aug 18, 2026
@kcreddy kcreddy added enhancement New feature or request Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Aug 18, 2026
@kcreddy
kcreddy marked this pull request as ready for review August 18, 2026 18:25
Copilot AI lite review requested due to automatic review settings August 18, 2026 18:25
@kcreddy
kcreddy requested review from a team as code owners August 18, 2026 18:25
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@kcreddy
kcreddy requested review from a team as code owners August 18, 2026 18:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

packages/amazon_security_lake/data_stream/event/elasticsearch/ingest_pipeline/pipeline_object_malware.yml: ASCII text, with CRLF line terminators

Comment thread packages/authentik/changelog.yml Outdated
Comment thread packages/1password/changelog.yml Outdated
Comment thread packages/azure/data_stream/signinlogs/elasticsearch/ingest_pipeline/default.yml Outdated
Comment thread packages/azure/data_stream/signinlogs/elasticsearch/ingest_pipeline/default.yml Outdated
Comment thread packages/claroty_ctd/data_stream/event/elasticsearch/ingest_pipeline/default.yml Outdated
Fix twelve inline comments from the PR review:

- Changelogs for 35 packages incorrectly claimed preserve_original_event
  was added; those packages already had it before this series. Description
  shortened to "Add tags to ingest pipeline processors." for packages where
  nothing was added to the on_failure block.

- carbon_black_cloud/watchlist_hit: the append_related_hosts_6d9a788e
  processor appended user.domain but was gated on ctx.host?.hostname;
  guard corrected to ctx.user?.domain, matching alert_v7 and endpoint_event.

- azure/signinlogs: tag azure-json-keys-to-snake-case renamed to
  azure_json_keys_to_snake_case (hyphens to underscores).

- azure/signinlogs, darktrace/ai_analyst_alert, darktrace/model_breach_alert
  (three scripts): single-line Painless sources reformatted with source: |-
  for readability.

- amazon_security_lake/pipeline_category_discovery.yml: add missing
  trailing newline.

- claroty_ctd/event: tag script_to_drop_N/A_values_fields renamed to
  script_to_drop_NA_values_fields (slash was not a valid identifier char).

Also update fix_pipeline_tags.py --normalize to convert hyphens to
underscores in tag stems (same logic as the asterisk-stripping pass).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 19, 2026 17:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

Copilot AI review requested due to automatic review settings August 19, 2026 19:04
@kcreddy
kcreddy requested a review from efd6 August 19, 2026 19:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@vera-review-bot

Copy link
Copy Markdown

🟢 Reviewed the latest commits 987c7d1…bb93b22 (46 commits) — nothing new beyond already posted comments.

Review summary

Issues found across earlier commits 7ae0c0b — 1 medium
  • 🟡 This append writes user.domain into related.hosts but is gated on host.hostname, so it appends an empty string whenever the hostname is set and the user domain is not (link) (Resolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package beyondtrust_pra - 1.3.0 containing this change is available at https://epr.elastic.co/package/beyondtrust_pra/1.3.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package bitdefender - 2.13.0 containing this change is available at https://epr.elastic.co/package/bitdefender/2.13.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package bitwarden - 1.22.0 containing this change is available at https://epr.elastic.co/package/bitwarden/1.22.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package blacklens - 1.4.0 containing this change is available at https://epr.elastic.co/package/blacklens/1.4.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package canva - 1.1.0 containing this change is available at https://epr.elastic.co/package/canva/1.1.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package carbon_black_cloud - 4.5.0 containing this change is available at https://epr.elastic.co/package/carbon_black_cloud/4.5.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package carbonblack_edr - 1.22.0 containing this change is available at https://epr.elastic.co/package/carbonblack_edr/1.22.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package checkpoint_email - 1.7.0 containing this change is available at https://epr.elastic.co/package/checkpoint_email/1.7.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package checkpoint_harmony_endpoint - 1.5.0 containing this change is available at https://epr.elastic.co/package/checkpoint_harmony_endpoint/1.5.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package cisa_kevs - 1.11.0 containing this change is available at https://epr.elastic.co/package/cisa_kevs/1.11.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package cisco_duo - 2.13.0 containing this change is available at https://epr.elastic.co/package/cisco_duo/2.13.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package cisco_secure_endpoint - 2.37.0 containing this change is available at https://epr.elastic.co/package/cisco_secure_endpoint/2.37.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package claroty_ctd - 1.6.0 containing this change is available at https://epr.elastic.co/package/claroty_ctd/1.6.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package claroty_xdome - 1.3.0 containing this change is available at https://epr.elastic.co/package/claroty_xdome/1.3.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package cloudflare - 2.37.0 containing this change is available at https://epr.elastic.co/package/cloudflare/2.37.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package crowdstrike - 4.8.0 containing this change is available at https://epr.elastic.co/package/crowdstrike/4.8.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package cyberark_epm - 1.6.0 containing this change is available at https://epr.elastic.co/package/cyberark_epm/1.6.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package cyberark_pta - 1.16.0 containing this change is available at https://epr.elastic.co/package/cyberark_pta/1.16.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package cyberarkpas - 2.29.0 containing this change is available at https://epr.elastic.co/package/cyberarkpas/2.29.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package cybereason - 1.9.0 containing this change is available at https://epr.elastic.co/package/cybereason/1.9.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package cylance - 0.25.0 containing this change is available at https://epr.elastic.co/package/cylance/0.25.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package darktrace - 2.4.0 containing this change is available at https://epr.elastic.co/package/darktrace/2.4.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package dataminr_pulse - 0.3.0 containing this change is available at https://epr.elastic.co/package/dataminr_pulse/0.3.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package digital_guardian - 1.12.0 containing this change is available at https://epr.elastic.co/package/digital_guardian/1.12.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package doppel - 1.1.0 containing this change is available at https://epr.elastic.co/package/doppel/1.1.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package doppler - 0.2.0 containing this change is available at https://epr.elastic.co/package/doppler/0.2.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package entityanalytics_ad - 0.23.0 containing this change is available at https://epr.elastic.co/package/entityanalytics_ad/0.23.0/

@elastic-vault-github-plugin-prod

Copy link
Copy Markdown
Contributor

Package entityanalytics_entra_id - 1.14.0 containing this change is available at https://epr.elastic.co/package/entityanalytics_entra_id/1.14.0/

seanrathier added a commit to seanrathier/integrations that referenced this pull request Aug 20, 2026
format_version 3.6.4 is required for provider_permissions, which was
added to the SecurityHub HTTPJSON input in this PR. The version bump
reflects the new 7.4.0 changelog entry (7.3.0 is already taken by
the processor-tags PR elastic#20572 that merged to main concurrently).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
seanrathier added a commit to seanrathier/integrations that referenced this pull request Aug 25, 2026
format_version 3.6.4 is required for provider_permissions, which was
added to the SecurityHub HTTPJSON input in this PR. The version bump
reflects the new 7.4.0 changelog entry (7.3.0 is already taken by
the processor-tags PR elastic#20572 that merged to main concurrently).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants