Skip to content

ssi_all: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error" - #12046

Merged
efd6 merged 0 commit into
elastic:mainfrom
efd6:12045-ssi-all
Dec 11, 2024
Merged

ssi_all: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error"#12046
efd6 merged 0 commit into
elastic:mainfrom
efd6:12045-ssi-all

Conversation

@efd6

@efd6 efd6 commented Dec 10, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed commit message

ssi_all: add "preserve_original_event" tag to documents with event.kind set to "pipeline_error"

Omit problematic packages: google_workspace, jamf_protect and ti_mandiant_advantage.

[git-generate]
for f in $(
    (
        for p in $(
            yq 'select(.owner.github == "elastic/security-service-integrations")|.name' packages/**/manifest.yml \
            | grep -v -- ---
        ); do
            rg -l -g 'default.yml' "value: pipeline_error" packages/$p
        done
    )|sort|uniq|egrep -v 'google_workspace|jamf_protect|ti_mandiant_advantage'
); do
    (grep 'value: preserve_original_event' $f >/dev/null 2>&1) && continue
    perl -i -pe 'BEGIN{undef $/;} s/([a-z:"])
( *)(- set:.*value: pipeline_error)/$1
$2$3
$2- append:
$2    field: tags
$2    value: preserve_original_event
$2    allow_duplicates: false/smg' $f
done
for p in $(git diff --name-only HEAD~1|cut -d/ -f1,2|sort|uniq); do
    (
        cd $p
        elastic-package changelog add \
            --description 'Add "preserve_original_event" tag to documents with `event.kind` set to "pipeline_error".' \
            --type enhancement \
            --next minor \
            --link https://github.com/elastic/integrations/pull/12046
    )>/dev/null 2>&1
done

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@efd6 efd6 added enhancement New feature or request Integration:All Bulk changes that touch every integration Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Dec 10, 2024
@efd6 efd6 self-assigned this Dec 10, 2024
@andrewkroh

andrewkroh commented Dec 10, 2024

Copy link
Copy Markdown
Contributor

I think we will want allow_duplicates: false on that append processor.

@efd6

efd6 commented Dec 10, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Yeah, it's not just that. There is some weirdness with the perl mutation that is brittle. I'm trying to fix other parts to see if that will fix this (I'm not optimistic).

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 10, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

Package tines 👍(0) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
time_saved 35714.29 25000 -10714.29 (-30%) 💔

To see the full report comment with /test benchmark fullreport

@efd6
efd6 marked this pull request as ready for review December 10, 2024 08:12
@efd6
efd6 requested review from a team as code owners December 10, 2024 08:12
@elasticmachine

elasticmachine commented Dec 10, 2024

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package ti_eclecticiq - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_eclecticiq/1.3.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package ti_eset - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_eset/1.3.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package ti_maltiverse - 1.3.0 containing this change is available at https://epr.elastic.co/package/ti_maltiverse/1.3.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package ti_misp - 1.36.0 containing this change is available at https://epr.elastic.co/package/ti_misp/1.36.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package ti_opencti - 2.4.0 containing this change is available at https://epr.elastic.co/package/ti_opencti/2.4.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package ti_otx - 1.26.0 containing this change is available at https://epr.elastic.co/package/ti_otx/1.26.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package ti_rapid7_threat_command - 2.1.0 containing this change is available at https://epr.elastic.co/package/ti_rapid7_threat_command/2.1.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package ti_recordedfuture - 1.27.0 containing this change is available at https://epr.elastic.co/package/ti_recordedfuture/1.27.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package ti_threatconnect - 1.4.0 containing this change is available at https://epr.elastic.co/package/ti_threatconnect/1.4.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package ti_threatq - 1.29.0 containing this change is available at https://epr.elastic.co/package/ti_threatq/1.29.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package tines - 1.13.0 containing this change is available at https://epr.elastic.co/package/tines/1.13.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package trellix_edr_cloud - 1.3.0 containing this change is available at https://epr.elastic.co/package/trellix_edr_cloud/1.3.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package trellix_epo_cloud - 1.12.0 containing this change is available at https://epr.elastic.co/package/trellix_epo_cloud/1.12.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package trend_micro_vision_one - 1.22.0 containing this change is available at https://epr.elastic.co/package/trend_micro_vision_one/1.22.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package trendmicro - 2.4.0 containing this change is available at https://epr.elastic.co/package/trendmicro/2.4.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package tychon - 0.2.0 containing this change is available at https://epr.elastic.co/package/tychon/0.2.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package vectra_detect - 1.10.0 containing this change is available at https://epr.elastic.co/package/vectra_detect/1.10.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package wiz - 2.5.0 containing this change is available at https://epr.elastic.co/package/wiz/2.5.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package zerofox - 1.26.0 containing this change is available at https://epr.elastic.co/package/zerofox/1.26.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package zeronetworks - 1.16.0 containing this change is available at https://epr.elastic.co/package/zeronetworks/1.16.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package zoom - 1.21.0 containing this change is available at https://epr.elastic.co/package/zoom/1.21.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package zscaler_zia - 3.5.0 containing this change is available at https://epr.elastic.co/package/zscaler_zia/3.5.0/

@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Dec 11, 2024

Copy link
Copy Markdown
Contributor

Package zscaler_zpa - 1.19.0 containing this change is available at https://epr.elastic.co/package/zscaler_zpa/1.19.0/

This was referenced Dec 11, 2024
Merged
Merged
Merged
@andrewkroh andrewkroh mentioned this pull request Sep 17, 2025
Closed
1 task
@github-actions github-actions Bot mentioned this pull request May 4, 2026
Open
This was referenced May 28, 2026
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kcreddy kcreddy kcreddy approved these changes
@shmsr shmsr shmsr approved these changes
@ishleenk17 ishleenk17 ishleenk17 approved these changes

Assignees

@efd6 efd6

Labels

enhancement New feature or request Integration:abnormal_security Abnormal AI Integration:akamai Akamai Integration:amazon_security_lake Amazon Security Lake Integration:atlassian_bitbucket Atlassian Bitbucket Integration:atlassian_confluence Atlassian Confluence Integration:atlassian_jira Atlassian Jira Integration:auth0 Auth0 Integration:authentik authentik Integration:aws_bedrock Amazon Bedrock Integration:azure_frontdoor Azure Frontdoor Integration:azure_network_watcher_nsg Azure Network Watcher NSG Integration:azure_network_watcher_vnet Azure Network Watcher VNet Integration:barracuda_cloudgen_firewall Barracuda CloudGen Firewall Logs Integration:barracuda Barracuda Web Application Firewall Integration:bbot BBOT (Bighuge BLS OSINT Tool) (Community supported) Integration:bitdefender BitDefender Integration:bitwarden Bitwarden Integration:blacklens blacklens.io (Community supported) Integration:box_events Box Events Integration:canva Canva Integration:carbon_black_cloud VMware Carbon Black Cloud Integration:carbonblack_edr VMware Carbon Black EDR Integration:checkpoint_email Check Point Harmony Email & Collaboration Integration:checkpoint_harmony_endpoint Check Point Harmony Endpoint Integration:cisa_kevs CISA Known Exploited Vulnerabilities (Community supported) Integration:cisco_duo Cisco Duo Integration:cisco_meraki Cisco Meraki Integration:cisco_secure_endpoint Cisco Secure Endpoint Integration:cisco_umbrella Cisco Umbrella Integration:claroty_ctd Claroty CTD Integration:cloudflare_logpush Cloudflare Logpush Integration:cloudflare Cloudflare Integration:crowdstrike CrowdStrike Integration:cyberark_pta Cyberark Privileged Threat Analytics Integration:cyberarkpas CyberArk Privileged Access Security Integration:cybereason Cybereason Integration:cylance CylanceProtect Logs (Deprecated) Integration:darktrace Darktrace Integration:digital_guardian Digital Guardian Integration:entityanalytics_ad Active Directory Entity Analytics Integration:entityanalytics_entra_id Microsoft Entra ID Entity Analytics Integration:entityanalytics_okta Okta Entity Analytics Integration:eset_protect ESET PROTECT Integration:f5_bigip F5 BIG-IP Integration:f5 F5 Logs (Deprecated) [Integration not found in source] Integration:1password 1Password Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@efd6 @andrewkroh @elasticmachine @kcreddy @shmsr @ishleenk17