Skip to content

Commit f32be99

Browse files
kcreddykcreddy
committed
ssi: add processor tags and preserve_original_event on failure (3/4)
Tag every ingest pipeline processor across 52 SSI-owned packages (servicenow through zscaler_zpa) with a `<name>_<content hash>` identifier, so that failure telemetry can attribute an error to the step that produced it rather than collapsing same-type processors into one bucket. All 20606 processors across the 265 pipeline files now carry a tag, including those nested inside `on_failure` handlers and `foreach` bodies, and those in the pipeline-level `on_failure` block. The hash is taken over the processor's content together with every processor enclosing it, with `tag` keys stripped. This makes uniqueness mechanical rather than positional, and gives identical constructs the same identity in every package, so a single query aggregates failures for one step across the whole estate; the hash carries that identity, the readable prefix is for humans. Add preserve_original_event to pipeline-level on_failure handlers that were missing it, ensuring the raw payload is retained when a pipeline error document is indexed. Two fixes ride along, each recorded in the affected package's changelog: - wiz: the pipeline-level `on_failure` handlers for the two cloud_configuration_finding data streams set `error.message` instead of appending to it, discarding what the processor-level handlers had already recorded. - ti_eset: the same `set` pattern in seven pipelines. Those record no processor-level errors, so this is a consistency fix with no change in behaviour. Ten processors that were exact duplicates of an earlier sibling, and so could never have any effect, are removed rather than given a disambiguating tag. These are enhancements, so the packages take a minor version bump. Updates elastic#20558
1 parent 59660e0 commit f32be99

369 files changed

Lines changed: 20725 additions & 20640 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎packages/servicenow/changelog.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# newer versions go on top
2-
- version: "2.1.2"
2+
- version: "2.2.0"
33
changes:
44
- description: Add tags to ingest pipeline processors and add `preserve_original_event` to pipeline-level `on_failure` handlers.
55
type: enhancement

‎packages/servicenow/data_stream/event/elasticsearch/ingest_pipeline/default.yml‎

Lines changed: 645 additions & 645 deletions
Large diffs are not rendered by default.

‎packages/servicenow/data_stream/event/elasticsearch/ingest_pipeline/pipeline_alm_hardware.yml‎

Lines changed: 71 additions & 71 deletions
Large diffs are not rendered by default.

‎packages/servicenow/data_stream/event/elasticsearch/ingest_pipeline/pipeline_change_request.yml‎

Lines changed: 88 additions & 88 deletions
Large diffs are not rendered by default.

‎packages/servicenow/data_stream/event/elasticsearch/ingest_pipeline/pipeline_cmdb_ci_business_app.yml‎

Lines changed: 41 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -3,57 +3,57 @@ description: Pipeline for processing Business Application logs.
33
processors:
44
- set:
55
field: event.kind
6-
tag: set_event_kind_to_event_cmdb_ci_business_app
6+
tag: set_event_kind_to_event_cmdb_ci_business_app_d452ef2c
77
value: event
88
- set:
99
field: event.category
10-
tag: append_event_category_cmdb_ci_business_app
10+
tag: append_event_category_cmdb_ci_business_app_a46e8e56
1111
value: ['configuration', 'host']
1212
- convert:
1313
field: servicenow.event.active_user_count.value
14-
tag: convert_active_user_count_value_to_long
14+
tag: convert_active_user_count_value_to_long_efacb1c6
1515
type: long
1616
ignore_missing: true
1717
if: ctx.servicenow?.event?.active_user_count?.value != ''
1818
on_failure:
1919
- remove:
20-
tag: remove_servicenow_event_active_user_count_value_d3ff2511
20+
tag: remove_servicenow_event_active_user_count_value_c51e6d64
2121
field: servicenow.event.active_user_count.value
2222
- append:
23-
tag: append_error_message_fabc7b59
23+
tag: append_error_message_ab766f07
2424
field: error.message
2525
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
2626
- convert:
2727
field: servicenow.event.certified.display_value
28-
tag: convert_certified_display_value_to_boolean
28+
tag: convert_certified_display_value_to_boolean_1bdf6dae
2929
type: boolean
3030
ignore_missing: true
3131
if: ctx.servicenow?.event?.certified?.display_value != ''
3232
on_failure:
3333
- remove:
34-
tag: remove_servicenow_event_certified_display_value_6a227aff
34+
tag: remove_servicenow_event_certified_display_value_7c82e143
3535
field: servicenow.event.certified.display_value
3636
- append:
37-
tag: append_error_message_37651567
37+
tag: append_error_message_66cc41b0
3838
field: error.message
3939
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
4040
- convert:
4141
field: servicenow.event.certified.value
42-
tag: convert_certified_value_to_boolean
42+
tag: convert_certified_value_to_boolean_6d33cafd
4343
type: boolean
4444
ignore_missing: true
4545
if: ctx.servicenow?.event?.certified?.value != ''
4646
on_failure:
4747
- remove:
48-
tag: remove_servicenow_event_certified_value_07204cfc
48+
tag: remove_servicenow_event_certified_value_b9588039
4949
field: servicenow.event.certified.value
5050
- append:
51-
tag: append_error_message_d6d4d6b5
51+
tag: append_error_message_c45904af
5252
field: error.message
5353
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
5454
- date:
5555
field: servicenow.event.contract_end_date.display_value
56-
tag: date_contract_end_date_display_value
56+
tag: date_contract_end_date_display_value_26ad6685
5757
target_field: servicenow.event.contract_end_date.display_value
5858
formats:
5959
- yyyy-MM-dd H:mm:ss
@@ -76,15 +76,15 @@ processors:
7676
if: ctx.servicenow?.event?.contract_end_date?.display_value != null && ctx.servicenow.event.contract_end_date.display_value != '' && ctx._conf?.date_format_preference == "ddMM"
7777
on_failure:
7878
- remove:
79-
tag: remove_servicenow_event_contract_end_date_display_value_6bbd1a8f
79+
tag: remove_servicenow_event_contract_end_date_display_value_92e93a8c
8080
field: servicenow.event.contract_end_date.display_value
8181
- append:
82-
tag: append_error_message_1504fe13
82+
tag: append_error_message_5ce678bb
8383
field: error.message
8484
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
8585
- date:
8686
field: servicenow.event.contract_end_date.display_value
87-
tag: date_contract_end_date_display_value_faa36f71
87+
tag: date_contract_end_date_display_value_d307ee03
8888
target_field: servicenow.event.contract_end_date.display_value
8989
formats:
9090
- yyyy-MM-dd H:mm:ss
@@ -107,15 +107,15 @@ processors:
107107
if: ctx.servicenow?.event?.contract_end_date?.display_value != null && ctx.servicenow.event.contract_end_date.display_value != '' && ctx._conf?.date_format_preference == "MMdd"
108108
on_failure:
109109
- remove:
110-
tag: remove_servicenow_event_contract_end_date_display_value_9347644f
110+
tag: remove_servicenow_event_contract_end_date_display_value_6b62b016
111111
field: servicenow.event.contract_end_date.display_value
112112
- append:
113-
tag: append_error_message_b59e8e23
113+
tag: append_error_message_33869d2e
114114
field: error.message
115115
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
116116
- date:
117117
field: servicenow.event.contract_end_date.value
118-
tag: date_contract_end_date_value
118+
tag: date_contract_end_date_value_d581466b
119119
target_field: servicenow.event.contract_end_date.value
120120
formats:
121121
- yyyy-MM-dd H:mm:ss
@@ -138,15 +138,15 @@ processors:
138138
if: ctx.servicenow?.event?.contract_end_date?.value != null && ctx.servicenow.event.contract_end_date.value != '' && ctx._conf?.date_format_preference == "ddMM"
139139
on_failure:
140140
- remove:
141-
tag: remove_servicenow_event_contract_end_date_value_5487f1f8
141+
tag: remove_servicenow_event_contract_end_date_value_af166b18
142142
field: servicenow.event.contract_end_date.value
143143
- append:
144-
tag: append_error_message_f4a7e963
144+
tag: append_error_message_e541546e
145145
field: error.message
146146
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
147147
- date:
148148
field: servicenow.event.contract_end_date.value
149-
tag: date_contract_end_date_value_20eb6084
149+
tag: date_contract_end_date_value_86755264
150150
target_field: servicenow.event.contract_end_date.value
151151
formats:
152152
- yyyy-MM-dd H:mm:ss
@@ -168,15 +168,15 @@ processors:
168168
if: ctx.servicenow?.event?.contract_end_date?.value != null && ctx.servicenow.event.contract_end_date.value != '' && ctx._conf?.date_format_preference == "MMdd"
169169
on_failure:
170170
- remove:
171-
tag: remove_servicenow_event_contract_end_date_value_6e39e95d
171+
tag: remove_servicenow_event_contract_end_date_value_666f2ef1
172172
field: servicenow.event.contract_end_date.value
173173
- append:
174-
tag: append_error_message_dabc142e
174+
tag: append_error_message_17b9558c
175175
field: error.message
176176
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
177177
- date:
178178
field: servicenow.event.next_assessment_date.display_value
179-
tag: date_next_assessment_date_display_value
179+
tag: date_next_assessment_date_display_value_81e3a094
180180
target_field: servicenow.event.next_assessment_date.display_value
181181
formats:
182182
- yyyy-MM-dd H:mm:ss
@@ -199,15 +199,15 @@ processors:
199199
if: ctx.servicenow?.event?.next_assessment_date?.display_value != null && ctx.servicenow.event.next_assessment_date.display_value != '' && ctx._conf?.date_format_preference == "ddMM"
200200
on_failure:
201201
- remove:
202-
tag: remove_servicenow_event_next_assessment_date_display_value_9e7d281f
202+
tag: remove_servicenow_event_next_assessment_date_display_value_284b83a9
203203
field: servicenow.event.next_assessment_date.display_value
204204
- append:
205-
tag: append_error_message_b9ab3583
205+
tag: append_error_message_f0158a87
206206
field: error.message
207207
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
208208
- date:
209209
field: servicenow.event.next_assessment_date.display_value
210-
tag: date_next_assessment_date_display_value_94aead59
210+
tag: date_next_assessment_date_display_value_17a71a87
211211
target_field: servicenow.event.next_assessment_date.display_value
212212
formats:
213213
- yyyy-MM-dd H:mm:ss
@@ -230,15 +230,15 @@ processors:
230230
if: ctx.servicenow?.event?.next_assessment_date?.display_value != null && ctx.servicenow.event.next_assessment_date.display_value != '' && ctx._conf?.date_format_preference == "MMdd"
231231
on_failure:
232232
- remove:
233-
tag: remove_servicenow_event_next_assessment_date_display_value_1103f0af
233+
tag: remove_servicenow_event_next_assessment_date_display_value_4606452b
234234
field: servicenow.event.next_assessment_date.display_value
235235
- append:
236-
tag: append_error_message_cbd0ec63
236+
tag: append_error_message_c13b8595
237237
field: error.message
238238
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
239239
- date:
240240
field: servicenow.event.next_assessment_date.value
241-
tag: date_next_assessment_date_value
241+
tag: date_next_assessment_date_value_1b31fcc0
242242
target_field: servicenow.event.next_assessment_date.value
243243
formats:
244244
- yyyy-MM-dd H:mm:ss
@@ -261,15 +261,15 @@ processors:
261261
if: ctx.servicenow?.event?.next_assessment_date?.value != null && ctx.servicenow.event.next_assessment_date.value != '' && ctx._conf?.date_format_preference == "ddMM"
262262
on_failure:
263263
- remove:
264-
tag: remove_servicenow_event_next_assessment_date_value_6ab92baa
264+
tag: remove_servicenow_event_next_assessment_date_value_d38d8381
265265
field: servicenow.event.next_assessment_date.value
266266
- append:
267-
tag: append_error_message_d412883d
267+
tag: append_error_message_437ccaf7
268268
field: error.message
269269
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
270270
- date:
271271
field: servicenow.event.next_assessment_date.value
272-
tag: date_next_assessment_date_value_38d1e4ac
272+
tag: date_next_assessment_date_value_917f5d12
273273
target_field: servicenow.event.next_assessment_date.value
274274
formats:
275275
- yyyy-MM-dd H:mm:ss
@@ -291,29 +291,29 @@ processors:
291291
if: ctx.servicenow?.event?.next_assessment_date?.value != null && ctx.servicenow.event.next_assessment_date.value != '' && ctx._conf?.date_format_preference == "MMdd"
292292
on_failure:
293293
- remove:
294-
tag: remove_servicenow_event_next_assessment_date_value_ce338e51
294+
tag: remove_servicenow_event_next_assessment_date_value_1770cc89
295295
field: servicenow.event.next_assessment_date.value
296296
- append:
297-
tag: append_error_message_97231dc6
297+
tag: append_error_message_f19be0cc
298298
field: error.message
299299
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
300300
- convert:
301301
field: servicenow.event.organization_unit_count.value
302-
tag: convert_organization_unit_count_value_to_long
302+
tag: convert_organization_unit_count_value_to_long_f273fed2
303303
type: long
304304
ignore_missing: true
305305
if: ctx.servicenow?.event?.organization_unit_count?.value != ''
306306
on_failure:
307307
- remove:
308-
tag: remove_servicenow_event_organization_unit_count_value_bed47c0b
308+
tag: remove_servicenow_event_organization_unit_count_value_f38decce
309309
field: servicenow.event.organization_unit_count.value
310310
- append:
311-
tag: append_error_message_bf27887d
311+
tag: append_error_message_6b144607
312312
field: error.message
313313
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
314314
on_failure:
315315
- append:
316-
tag: append_error_message
316+
tag: append_error_message_d916120d
317317
field: error.message
318318
value: >-
319319
Processor '{{{ _ingest.on_failure_processor_type }}}'
@@ -323,10 +323,10 @@ on_failure:
323323
}}}'
324324
- set:
325325
field: event.kind
326-
tag: set_pipeline_error_to_event_kind
326+
tag: set_pipeline_error_to_event_kind_72b1902f
327327
value: pipeline_error
328328
- append:
329-
tag: append_tags
329+
tag: append_tags_279d5a5c
330330
field: tags
331331
value: preserve_original_event
332332
allow_duplicates: false

0 commit comments

Comments
 (0)