Repository navigation
add processor tags and preserve_original_event on failure (1/4) - #20572
Conversation
Tag every ingest pipeline processor across 51 SSI-owned packages (1password through entityanalytics_entra_id) with a unique, descriptive identifier so that failure telemetry can attribute errors to the specific step that failed rather than collapsing same-type processors into one bucket. Add preserve_original_event to pipeline-level on_failure handlers that were missing it, ensuring the raw payload is retained when a pipeline error document is indexed. Updates elastic#20558
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
Update the link field in each new changelog entry from the tracking issue to the merged pull request. Updates elastic#20558
Tag every ingest pipeline processor across 51 SSI-owned packages
(1password through entityanalytics_entra_id), so that failure telemetry
can attribute an error to the step that produced it rather than
collapsing same-type processors into one bucket. All 25752 processors
across the 303 pipeline files now carry a tag, 10853 of them newly,
including those nested inside `on_failure` handlers and `foreach`
bodies, and those in the pipeline-level `on_failure` block.
Existing tags keep their name. Where a name was not already unique in
its pipeline, or the processor had no tag at all, it gains an 8-hex
suffix hashed over the processor's content and everything enclosing it.
Tags that already carried a hash are left byte-identical -- 3275 of
them, in aws, crowdstrike and axonius, which shipped tags before this
series; a hash is only meaningful relative to the generator that
produced it, so re-hashing a published tag churns a value consumers may
key off. Those three packages therefore keep the earlier suffix format.
Tags that would have said nothing (`script_<hash>`, `fail_<hash>`) are
seeded from the processor's description, name, message or preceding
comment, and tags whose leading word named the wrong action (`set_...`
on an append) are corrected.
Add preserve_original_event to pipeline-level on_failure handlers that
were missing it, ensuring the raw payload is retained when a pipeline
error document is indexed. Sixteen of the 51 packages needed it.
Six processors across five packages were exact duplicates of an earlier
sibling and could never have any effect; they are removed rather than
given a disambiguating tag. Two further fixes ride along, each in the
affected package's changelog:
- aws, aws_bedrock: five pipeline-level on_failure handlers set
error.message instead of appending, discarding what processor-level
handlers recorded. error.message is now an array on failed
documents.
- azure: the aadgraphactivitylogs link to the shared pipeline is
refreshed, since tagging that pipeline changed its checksum.
These are enhancements, so the packages take a minor version bump.
Updates elastic#20558
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
entity |
8928.57 | 7092.2 | -1836.37 (-20.57%) | 💔 |
To see the full report comment with /test benchmark fullreport
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
There was a problem hiding this comment.
packages/amazon_security_lake/data_stream/event/elasticsearch/ingest_pipeline/pipeline_object_malware.yml: ASCII text, with CRLF line terminators
Fix twelve inline comments from the PR review: - Changelogs for 35 packages incorrectly claimed preserve_original_event was added; those packages already had it before this series. Description shortened to "Add tags to ingest pipeline processors." for packages where nothing was added to the on_failure block. - carbon_black_cloud/watchlist_hit: the append_related_hosts_6d9a788e processor appended user.domain but was gated on ctx.host?.hostname; guard corrected to ctx.user?.domain, matching alert_v7 and endpoint_event. - azure/signinlogs: tag azure-json-keys-to-snake-case renamed to azure_json_keys_to_snake_case (hyphens to underscores). - azure/signinlogs, darktrace/ai_analyst_alert, darktrace/model_breach_alert (three scripts): single-line Painless sources reformatted with source: |- for readability. - amazon_security_lake/pipeline_category_discovery.yml: add missing trailing newline. - claroty_ctd/event: tag script_to_drop_N/A_values_fields renamed to script_to_drop_NA_values_fields (slash was not a valid identifier char). Also update fix_pipeline_tags.py --normalize to convert hyphens to underscores in tag stems (same logic as the asterisk-stripping pass). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
🟢 Reviewed the latest commits 987c7d1…bb93b22 (46 commits) — nothing new beyond already posted comments. Review summaryIssues found across earlier commits 7ae0c0b — 1 medium
🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills
|
|
Package beyondtrust_pra - 1.3.0 containing this change is available at https://epr.elastic.co/package/beyondtrust_pra/1.3.0/ |
|
Package bitdefender - 2.13.0 containing this change is available at https://epr.elastic.co/package/bitdefender/2.13.0/ |
|
Package bitwarden - 1.22.0 containing this change is available at https://epr.elastic.co/package/bitwarden/1.22.0/ |
|
Package blacklens - 1.4.0 containing this change is available at https://epr.elastic.co/package/blacklens/1.4.0/ |
|
Package canva - 1.1.0 containing this change is available at https://epr.elastic.co/package/canva/1.1.0/ |
|
Package carbon_black_cloud - 4.5.0 containing this change is available at https://epr.elastic.co/package/carbon_black_cloud/4.5.0/ |
|
Package carbonblack_edr - 1.22.0 containing this change is available at https://epr.elastic.co/package/carbonblack_edr/1.22.0/ |
|
Package checkpoint_email - 1.7.0 containing this change is available at https://epr.elastic.co/package/checkpoint_email/1.7.0/ |
|
Package checkpoint_harmony_endpoint - 1.5.0 containing this change is available at https://epr.elastic.co/package/checkpoint_harmony_endpoint/1.5.0/ |
|
Package cisa_kevs - 1.11.0 containing this change is available at https://epr.elastic.co/package/cisa_kevs/1.11.0/ |
|
Package cisco_duo - 2.13.0 containing this change is available at https://epr.elastic.co/package/cisco_duo/2.13.0/ |
|
Package cisco_secure_endpoint - 2.37.0 containing this change is available at https://epr.elastic.co/package/cisco_secure_endpoint/2.37.0/ |
|
Package claroty_ctd - 1.6.0 containing this change is available at https://epr.elastic.co/package/claroty_ctd/1.6.0/ |
|
Package claroty_xdome - 1.3.0 containing this change is available at https://epr.elastic.co/package/claroty_xdome/1.3.0/ |
|
Package cloudflare - 2.37.0 containing this change is available at https://epr.elastic.co/package/cloudflare/2.37.0/ |
|
Package crowdstrike - 4.8.0 containing this change is available at https://epr.elastic.co/package/crowdstrike/4.8.0/ |
|
Package cyberark_epm - 1.6.0 containing this change is available at https://epr.elastic.co/package/cyberark_epm/1.6.0/ |
|
Package cyberark_pta - 1.16.0 containing this change is available at https://epr.elastic.co/package/cyberark_pta/1.16.0/ |
|
Package cyberarkpas - 2.29.0 containing this change is available at https://epr.elastic.co/package/cyberarkpas/2.29.0/ |
|
Package cybereason - 1.9.0 containing this change is available at https://epr.elastic.co/package/cybereason/1.9.0/ |
|
Package cylance - 0.25.0 containing this change is available at https://epr.elastic.co/package/cylance/0.25.0/ |
|
Package darktrace - 2.4.0 containing this change is available at https://epr.elastic.co/package/darktrace/2.4.0/ |
|
Package dataminr_pulse - 0.3.0 containing this change is available at https://epr.elastic.co/package/dataminr_pulse/0.3.0/ |
|
Package digital_guardian - 1.12.0 containing this change is available at https://epr.elastic.co/package/digital_guardian/1.12.0/ |
|
Package doppel - 1.1.0 containing this change is available at https://epr.elastic.co/package/doppel/1.1.0/ |
|
Package doppler - 0.2.0 containing this change is available at https://epr.elastic.co/package/doppler/0.2.0/ |
|
Package entityanalytics_ad - 0.23.0 containing this change is available at https://epr.elastic.co/package/entityanalytics_ad/0.23.0/ |
|
Package entityanalytics_entra_id - 1.14.0 containing this change is available at https://epr.elastic.co/package/entityanalytics_entra_id/1.14.0/ |
format_version 3.6.4 is required for provider_permissions, which was added to the SecurityHub HTTPJSON input in this PR. The version bump reflects the new 7.4.0 changelog entry (7.3.0 is already taken by the processor-tags PR elastic#20572 that merged to main concurrently). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
format_version 3.6.4 is required for provider_permissions, which was added to the SecurityHub HTTPJSON input in this PR. The version bump reflects the new 7.4.0 changelog entry (7.3.0 is already taken by the processor-tags PR elastic#20572 that merged to main concurrently). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Proposed commit message
Checklist
changelog.ymlfile.Related issues