Skip to content

ssi: add processor tags and preserve_original_event on failure (3/4) #20574

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kcreddy merged 12 commits into from
Aug 18, 2026
Merged

ssi: add processor tags and preserve_original_event on failure (3/4) #20574

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
101b6ad
ssi: add processor tags and preserve_original_event on failure (3/4)
kcreddy Aug 6, 2026
ccd385c
ssi: update changelog links to PR #20574
kcreddy Aug 6, 2026
d0967cb
ssi: add processor tags and preserve_original_event on failure (3/4)
kcreddy Aug 11, 2026
4faac5d
Merge remote-tracking branch 'upstream/main' into tagging-evt-origina…
kcreddy Aug 11, 2026
59660e0
Merge remote-tracking branch 'upstream/main' into tagging-evt-origina…
kcreddy Aug 12, 2026
f32be99
ssi: add processor tags and preserve_original_event on failure (3/4)
kcreddy Aug 12, 2026
239f005
Address comments from -4
kcreddy Aug 12, 2026
8dfd3e7
Address event.original removal and fix pipeline tests
kcreddy Aug 12, 2026
eb4f36b
Address review comments
kcreddy Aug 13, 2026
74b4080
resolve comments
kcreddy Aug 13, 2026
15fe22f
Address review comments
kcreddy Aug 17, 2026
8b3c820
Merge remote-tracking branch 'upstream/main' into tagging-evt-origina…
kcreddy Aug 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address event.original removal and fix pipeline tests
  • Loading branch information
@kcreddy
kcreddy committed Aug 12, 2026
commit 8dfd3e7c6ebd1b56e51fc7e7cb32baa6fbdbc773
3 changes: 3 additions & 0 deletions packages/ti_socradar_taxii/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
- description: Add tags to ingest pipeline processors and add `preserve_original_event` to pipeline-level `on_failure` handlers.
type: enhancement
link: https://github.com/elastic/integrations/pull/20574
- description: Stop removing `event.original` in the `indicator` data stream. It was removed before `preserve_original_event` was added to `tags`, so `pipeline_error` documents never retained the original payload.
type: bugfix
link: https://github.com/elastic/integrations/pull/20574
- version: "0.2.2"
changes:
- description: Set agentless deployment mode `release` field to `ga`.
Expand Down
1 change: 1 addition & 0 deletions ...radar_taxii/data_stream/indicator/_dev/test/pipeline/test-indicator-asn.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-111111111106",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-111111111106\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[autonomous-system:number = 4294967294]\",\"pattern_type\":\"stix\",\"confidence\":60,\"labels\":[\"malicious-activity\"],\"name\":\"Malicious ASN (large 32-bit)\"}",
"type": [
"indicator"
]
Expand Down
1 change: 1 addition & 0 deletions ...ar_taxii/data_stream/indicator/_dev/test/pipeline/test-indicator-domain.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-111111111102",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-111111111102\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[domain-name:value = 'malicious-example.com']\",\"pattern_type\":\"stix\",\"confidence\":50,\"labels\":[\"malicious-activity\"],\"name\":\"Malicious Domain\"}",
"type": [
"indicator"
]
Expand Down
2 changes: 2 additions & 0 deletions ...dar_taxii/data_stream/indicator/_dev/test/pipeline/test-indicator-email.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-111111111104",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-111111111104\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[email-addr:value = 'phisher@evil-example.com']\",\"pattern_type\":\"stix\",\"confidence\":30,\"labels\":[\"malicious-activity\"],\"name\":\"Phishing Email\"}",
"type": [
"indicator"
]
Expand Down Expand Up @@ -70,6 +71,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-11111111110e",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-11111111110e\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[email-message:from_ref.value = 'spammer@evil-example.com']\",\"pattern_type\":\"stix\",\"confidence\":40,\"labels\":[\"malicious-activity\"],\"name\":\"Spam Email Message\"}",
"type": [
"indicator"
]
Expand Down
4 changes: 4 additions & 0 deletions ...adar_taxii/data_stream/indicator/_dev/test/pipeline/test-indicator-file.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-111111111105",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-111111111105\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[file:hashes.'SHA-256' = 'd7a8fbb307d7809469ca5ce888b23a65f06e4563cc956e73e82f5c72c865aab6']\",\"pattern_type\":\"stix\",\"confidence\":90,\"labels\":[\"malicious-activity\"],\"name\":\"Malicious File SHA-256\"}",
"type": [
"indicator"
]
Expand Down Expand Up @@ -77,6 +78,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-11111111110b",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-11111111110b\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[file:hashes.'MD5' = 'd41d8cd98f00b204e9800998ecf8427e']\",\"pattern_type\":\"stix\",\"confidence\":60,\"labels\":[\"malicious-activity\"],\"name\":\"Malicious File MD5\"}",
"type": [
"indicator"
]
Expand Down Expand Up @@ -142,6 +144,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-11111111110c",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-11111111110c\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[file:hashes.'SHA-384' = '38b060a751ac96384cd9327eb1b1e36a21fdb71114be07434c0cc7bf63f6e1da274edebfe76f65fbd51ad2f14898b95b']\",\"pattern_type\":\"stix\",\"confidence\":85,\"labels\":[\"malicious-activity\"],\"name\":\"Malicious File SHA-384\"}",
"type": [
"indicator"
]
Expand Down Expand Up @@ -207,6 +210,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-11111111110d",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-11111111110d\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[file:hashes.'SHA-512' = 'cf83e1357eefb8bdf1542850d66d8007d620e4050b5715dc83f4a921d36ce9ce47d0d13c5d85f2b0ff8318d2877eec2f63b931bd47417a81a538327af927da3e']\",\"pattern_type\":\"stix\",\"confidence\":80,\"labels\":[\"malicious-activity\"],\"name\":\"Malicious File SHA-512\"}",
"type": [
"indicator"
]
Expand Down
2 changes: 2 additions & 0 deletions ...cradar_taxii/data_stream/indicator/_dev/test/pipeline/test-indicator-ip.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-111111111101",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-111111111101\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"valid_until\":\"2026-08-17T12:00:00.000Z\",\"pattern\":\"[ipv4-addr:value = '203.0.113.50']\",\"pattern_type\":\"stix\",\"confidence\":75,\"labels\":[\"malicious-activity\"],\"name\":\"Malicious IPv4\",\"description\":\"Test IP indicator\"}",
"type": [
"indicator"
]
Expand Down Expand Up @@ -75,6 +76,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-11111111110a",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-11111111110a\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[ipv6-addr:value = '2001:db8::1']\",\"pattern_type\":\"stix\",\"confidence\":50,\"labels\":[\"malicious-activity\"],\"name\":\"Malicious IPv6\"}",
"type": [
"indicator"
]
Expand Down
1 change: 1 addition & 0 deletions ...radar_taxii/data_stream/indicator/_dev/test/pipeline/test-indicator-url.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-111111111103",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-111111111103\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[url:value = 'https://malicious-example.com/payload']\",\"pattern_type\":\"stix\",\"confidence\":80,\"labels\":[\"malicious-activity\"],\"name\":\"Malicious URL\"}",
"type": [
"indicator"
]
Expand Down
1 change: 1 addition & 0 deletions ...ata_stream/indicator/_dev/test/pipeline/test-indicator-windows-registry.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-111111111107",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-111111111107\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[windows-registry-key:key = 'HKEY_LOCAL_MACHINE\\\\\\\\SOFTWARE\\\\\\\\Malware']\",\"pattern_type\":\"stix\",\"confidence\":40,\"labels\":[\"malicious-activity\"],\"name\":\"Malicious Registry Key\"}",
"type": [
"indicator"
]
Expand Down
2 changes: 2 additions & 0 deletions ...adar_taxii/data_stream/indicator/_dev/test/pipeline/test-indicator-x509.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-111111111108",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-111111111108\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[x509-certificate:hashes.'SHA-256' = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa']\",\"pattern_type\":\"stix\",\"confidence\":70,\"labels\":[\"malicious-activity\"],\"name\":\"Malicious X509\"}",
"type": [
"indicator"
]
Expand Down Expand Up @@ -77,6 +78,7 @@
"dataset": "ti_socradar_taxii.indicator",
"id": "indicator--11111111-1111-1111-1111-11111111110f",
"kind": "enrichment",
"original": "{\"type\":\"indicator\",\"spec_version\":\"2.1\",\"id\":\"indicator--11111111-1111-1111-1111-11111111110f\",\"created\":\"2026-02-17T12:00:00.000Z\",\"modified\":\"2026-02-17T12:00:00.000Z\",\"valid_from\":\"2026-02-17T12:00:00.000Z\",\"pattern\":\"[x509-certificate:serial_number = '01:23:45:67:89:ab:cd:ef']\",\"pattern_type\":\"stix\",\"confidence\":50,\"labels\":[\"malicious-activity\"],\"name\":\"X509 Serial\"}",
"type": [
"indicator"
]
Expand Down
6 changes: 0 additions & 6 deletions packages/ti_socradar_taxii/data_stream/indicator/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -476,12 +476,6 @@ processors:
}
}
if (ctx.event != null && saved_original != null) ctx.event['original'] = saved_original;
- remove:
field: event.original
tag: remove_event_original_0afd5d5d
if: ctx.tags == null || !ctx.tags.contains('preserve_original_event')
ignore_missing: true

- script:
tag: drop_null_empty_values_9d8aede1
lang: painless
Expand Down
Loading