Skip to content

ssi: add processor tags and preserve_original_event on failure (3/4) #20574

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kcreddy merged 12 commits into from
Aug 18, 2026
Merged

ssi: add processor tags and preserve_original_event on failure (3/4) #20574

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
101b6ad
ssi: add processor tags and preserve_original_event on failure (3/4)
kcreddy Aug 6, 2026
ccd385c
ssi: update changelog links to PR #20574
kcreddy Aug 6, 2026
d0967cb
ssi: add processor tags and preserve_original_event on failure (3/4)
kcreddy Aug 11, 2026
4faac5d
Merge remote-tracking branch 'upstream/main' into tagging-evt-origina…
kcreddy Aug 11, 2026
59660e0
Merge remote-tracking branch 'upstream/main' into tagging-evt-origina…
kcreddy Aug 12, 2026
f32be99
ssi: add processor tags and preserve_original_event on failure (3/4)
kcreddy Aug 12, 2026
239f005
Address comments from -4
kcreddy Aug 12, 2026
8dfd3e7
Address event.original removal and fix pipeline tests
kcreddy Aug 12, 2026
eb4f36b
Address review comments
kcreddy Aug 13, 2026
74b4080
resolve comments
kcreddy Aug 13, 2026
15fe22f
Address review comments
kcreddy Aug 17, 2026
8b3c820
Merge remote-tracking branch 'upstream/main' into tagging-evt-origina…
kcreddy Aug 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
5 changes: 5 additions & 0 deletions packages/servicenow/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top

Check notice on line 1 in packages/servicenow/changelog.yml

View workflow job for this annotation

GitHub Actions / Lint user-facing content 

Elastic.Versions: Use 'later versions' instead of 'newer versions' when referring to versions.
- version: "2.2.0"
changes:
- description: Add tags to ingest pipeline processors and add `preserve_original_event` to pipeline-level `on_failure` handlers.
type: enhancement
link: https://github.com/elastic/integrations/pull/20574
- version: "2.1.1"
changes:
- description: Set agentless deployment mode `release` field to `ga`.
Expand All @@ -16,7 +21,7 @@
link: https://github.com/elastic/integrations/pull/19257
- version: "2.0.0"
changes:
- description: Add configurable date format handling to eliminate ambiguous date parsing.

Check notice on line 24 in packages/servicenow/changelog.yml

View workflow job for this annotation

GitHub Actions / Lint user-facing content 

Elastic.Wordiness: Consider using 'remove' instead of 'eliminate'.
type: breaking-change
link: https://github.com/elastic/integrations/pull/17674
- version: "1.3.3"
Expand Down Expand Up @@ -51,7 +56,7 @@
link: https://github.com/elastic/integrations/pull/15602
- version: "1.1.0"
changes:
- description: Use `terminate` processor instead of `fail` processor to handle agent errors.

Check notice on line 59 in packages/servicenow/changelog.yml

View workflow job for this annotation

GitHub Actions / Lint user-facing content 

Elastic.WordChoice: Consider using 'stop, exit' instead of 'terminate', unless the term is in the UI.
type: enhancement
link: https://github.com/elastic/integrations/pull/14393
- version: "1.0.2"
Expand Down
923 changes: 647 additions & 276 deletions packages/servicenow/data_stream/event/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop

Large diffs are not rendered by default.

102 changes: 75 additions & 27 deletions ...ages/servicenow/data_stream/event/elasticsearch/ingest_pipeline/pipeline_alm_hardware.yml
View file
Open in desktop

Large diffs are not rendered by default.

128 changes: 93 additions & 35 deletions ...es/servicenow/data_stream/event/elasticsearch/ingest_pipeline/pipeline_change_request.yml
View file
Open in desktop

Large diffs are not rendered by default.

66 changes: 47 additions & 19 deletions ...vicenow/data_stream/event/elasticsearch/ingest_pipeline/pipeline_cmdb_ci_business_app.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,51 +3,57 @@ description: Pipeline for processing Business Application logs.
processors:
- set:
field: event.kind
tag: set_event_kind_to_event_cmdb_ci_business_app
tag: set_event_kind_to_event_cmdb_ci_business_app_d452ef2c
value: event
- set:
field: event.category
tag: append_event_category_cmdb_ci_business_app
tag: set_event_category_cmdb_ci_business_app_a46e8e56
value: ['configuration', 'host']
- convert:
field: servicenow.event.active_user_count.value
tag: convert_active_user_count_value_to_long
tag: convert_active_user_count_value_to_long_efacb1c6
type: long
ignore_missing: true
if: ctx.servicenow?.event?.active_user_count?.value != ''
on_failure:
- remove:
tag: remove_servicenow_event_active_user_count_value_c51e6d64
field: servicenow.event.active_user_count.value
- append:
tag: append_error_message_ab766f07
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- convert:
field: servicenow.event.certified.display_value
tag: convert_certified_display_value_to_boolean
tag: convert_certified_display_value_to_boolean_1bdf6dae
type: boolean
ignore_missing: true
if: ctx.servicenow?.event?.certified?.display_value != ''
on_failure:
- remove:
tag: remove_servicenow_event_certified_display_value_7c82e143
field: servicenow.event.certified.display_value
- append:
tag: append_error_message_66cc41b0
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- convert:
field: servicenow.event.certified.value
tag: convert_certified_value_to_boolean
tag: convert_certified_value_to_boolean_6d33cafd
type: boolean
ignore_missing: true
if: ctx.servicenow?.event?.certified?.value != ''
on_failure:
- remove:
tag: remove_servicenow_event_certified_value_b9588039
field: servicenow.event.certified.value
- append:
tag: append_error_message_c45904af
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- date:
field: servicenow.event.contract_end_date.display_value
tag: date_contract_end_date_display_value
tag: date_contract_end_date_display_value_26ad6685
target_field: servicenow.event.contract_end_date.display_value
formats:
- yyyy-MM-dd H:mm:ss
Expand All @@ -70,13 +76,15 @@ processors:
if: ctx.servicenow?.event?.contract_end_date?.display_value != null && ctx.servicenow.event.contract_end_date.display_value != '' && ctx._conf?.date_format_preference == "ddMM"
on_failure:
- remove:
tag: remove_servicenow_event_contract_end_date_display_value_92e93a8c
field: servicenow.event.contract_end_date.display_value
- append:
tag: append_error_message_5ce678bb
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- date:
field: servicenow.event.contract_end_date.display_value
tag: date_contract_end_date_display_value
tag: date_contract_end_date_display_value_d307ee03
target_field: servicenow.event.contract_end_date.display_value
formats:
- yyyy-MM-dd H:mm:ss
Expand All @@ -99,13 +107,15 @@ processors:
if: ctx.servicenow?.event?.contract_end_date?.display_value != null && ctx.servicenow.event.contract_end_date.display_value != '' && ctx._conf?.date_format_preference == "MMdd"
on_failure:
- remove:
tag: remove_servicenow_event_contract_end_date_display_value_6b62b016
field: servicenow.event.contract_end_date.display_value
- append:
tag: append_error_message_33869d2e
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- date:
field: servicenow.event.contract_end_date.value
tag: date_contract_end_date_value
tag: date_contract_end_date_value_d581466b
target_field: servicenow.event.contract_end_date.value
formats:
- yyyy-MM-dd H:mm:ss
Expand All @@ -128,13 +138,15 @@ processors:
if: ctx.servicenow?.event?.contract_end_date?.value != null && ctx.servicenow.event.contract_end_date.value != '' && ctx._conf?.date_format_preference == "ddMM"
on_failure:
- remove:
tag: remove_servicenow_event_contract_end_date_value_af166b18
field: servicenow.event.contract_end_date.value
- append:
tag: append_error_message_e541546e
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- date:
field: servicenow.event.contract_end_date.value
tag: date_contract_end_date_value
tag: date_contract_end_date_value_86755264
target_field: servicenow.event.contract_end_date.value
formats:
- yyyy-MM-dd H:mm:ss
Expand All @@ -156,13 +168,15 @@ processors:
if: ctx.servicenow?.event?.contract_end_date?.value != null && ctx.servicenow.event.contract_end_date.value != '' && ctx._conf?.date_format_preference == "MMdd"
on_failure:
- remove:
tag: remove_servicenow_event_contract_end_date_value_666f2ef1
field: servicenow.event.contract_end_date.value
- append:
tag: append_error_message_17b9558c
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- date:
field: servicenow.event.next_assessment_date.display_value
tag: date_next_assessment_date_display_value
tag: date_next_assessment_date_display_value_81e3a094
target_field: servicenow.event.next_assessment_date.display_value
formats:
- yyyy-MM-dd H:mm:ss
Expand All @@ -182,17 +196,18 @@ processors:
- dd.MM.yyyy H:mm:ss
- dd.MM.yyyy HH:mm:ss
- dd.MM.yyyy
if: ctx.servicenow?.event?.next_assessment_date?.display_value != null &&
ctx.servicenow.event.next_assessment_date.display_value != '' && ctx._conf?.date_format_preference == "ddMM"
if: ctx.servicenow?.event?.next_assessment_date?.display_value != null && ctx.servicenow.event.next_assessment_date.display_value != '' && ctx._conf?.date_format_preference == "ddMM"
on_failure:
- remove:
tag: remove_servicenow_event_next_assessment_date_display_value_284b83a9
field: servicenow.event.next_assessment_date.display_value
- append:
tag: append_error_message_f0158a87
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- date:
field: servicenow.event.next_assessment_date.display_value
tag: date_next_assessment_date_display_value
tag: date_next_assessment_date_display_value_17a71a87
target_field: servicenow.event.next_assessment_date.display_value
formats:
- yyyy-MM-dd H:mm:ss
Expand All @@ -212,17 +227,18 @@ processors:
- dd.MM.yyyy H:mm:ss
- dd.MM.yyyy HH:mm:ss
- dd.MM.yyyy
if: ctx.servicenow?.event?.next_assessment_date?.display_value != null &&
ctx.servicenow.event.next_assessment_date.display_value != '' && ctx._conf?.date_format_preference == "MMdd"
if: ctx.servicenow?.event?.next_assessment_date?.display_value != null && ctx.servicenow.event.next_assessment_date.display_value != '' && ctx._conf?.date_format_preference == "MMdd"
on_failure:
- remove:
tag: remove_servicenow_event_next_assessment_date_display_value_4606452b
field: servicenow.event.next_assessment_date.display_value
- append:
tag: append_error_message_c13b8595
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- date:
field: servicenow.event.next_assessment_date.value
tag: date_next_assessment_date_value
tag: date_next_assessment_date_value_1b31fcc0
target_field: servicenow.event.next_assessment_date.value
formats:
- yyyy-MM-dd H:mm:ss
Expand All @@ -245,13 +261,15 @@ processors:
if: ctx.servicenow?.event?.next_assessment_date?.value != null && ctx.servicenow.event.next_assessment_date.value != '' && ctx._conf?.date_format_preference == "ddMM"
on_failure:
- remove:
tag: remove_servicenow_event_next_assessment_date_value_d38d8381
field: servicenow.event.next_assessment_date.value
- append:
tag: append_error_message_437ccaf7
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- date:
field: servicenow.event.next_assessment_date.value
tag: date_next_assessment_date_value
tag: date_next_assessment_date_value_917f5d12
target_field: servicenow.event.next_assessment_date.value
formats:
- yyyy-MM-dd H:mm:ss
Expand All @@ -273,24 +291,29 @@ processors:
if: ctx.servicenow?.event?.next_assessment_date?.value != null && ctx.servicenow.event.next_assessment_date.value != '' && ctx._conf?.date_format_preference == "MMdd"
on_failure:
- remove:
tag: remove_servicenow_event_next_assessment_date_value_1770cc89
field: servicenow.event.next_assessment_date.value
- append:
tag: append_error_message_f19be0cc
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
- convert:
field: servicenow.event.organization_unit_count.value
tag: convert_organization_unit_count_value_to_long
tag: convert_organization_unit_count_value_to_long_f273fed2
type: long
ignore_missing: true
if: ctx.servicenow?.event?.organization_unit_count?.value != ''
on_failure:
- remove:
tag: remove_servicenow_event_organization_unit_count_value_f38decce
field: servicenow.event.organization_unit_count.value
- append:
tag: append_error_message_6b144607
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
on_failure:
- append:
tag: append_error_message_d916120d
field: error.message
value: >-
Processor '{{{ _ingest.on_failure_processor_type }}}'
Expand All @@ -300,5 +323,10 @@ on_failure:
}}}'
- set:
field: event.kind
tag: set_pipeline_error_to_event_kind
tag: set_pipeline_error_to_event_kind_72b1902f
value: pipeline_error
- append:
tag: append_tags_279d5a5c
field: tags
value: preserve_original_event
allow_duplicates: false
Loading
Loading