Skip to content

Commit ea3c186

Browse files
kcreddykcreddy
authored
ssi: add processor tags and preserve_original_event on failure (3/4) (#20574)
Tag every ingest pipeline processor across 52 SSI-owned packages (servicenow through zscaler_zpa), so that failure telemetry can attribute an error to the step that produced it rather than collapsing same-type processors into one bucket. All 20605 processors across the 265 pipeline files now carry a tag, 10045 of them newly, including those nested inside `on_failure` handlers and `foreach` bodies, and those in the pipeline-level `on_failure` block. Existing tags keep their name. Where a name was not already unique in its pipeline, or the processor had no tag at all, it gains an 8-hex suffix hashed over the processor's content and everything enclosing it. Tags that already carried a hash are left byte-identical -- 81 of them, in ti_anomali and zscaler_zia, which shipped tags before this series; a hash is only meaningful relative to the generator that produced it, so re-hashing a published tag churns a value consumers may key off. Those two packages therefore keep the earlier suffix format. Tags that would have said nothing (`script_<hash>`, `fail_<hash>`) are seeded from the processor's description, name, message or preceding comment, and tags whose leading word named the wrong action (`set_...` on an append) are corrected. Add preserve_original_event to pipeline-level on_failure handlers that were missing it, ensuring the raw payload is retained when a pipeline error document is indexed. Twelve of the 52 packages needed it. Ten processors across six packages were exact duplicates of an earlier sibling and could never have any effect; they are removed rather than given a disambiguating tag. Three further fixes ride along, each in the affected package's changelog: - ti_socradar_taxii: event.original was removed before preserve_original_event was added to tags, so pipeline_error documents never retained the payload. The removal is dropped and the test expectations regenerated. - wiz, ti_eset: pipeline-level on_failure set error.message instead of appending, discarding what processor-level handlers recorded. error.message is now an array on failed documents. These are enhancements, so the packages take a minor version bump. Updates #20558
1 parent 37cd5c6 commit ea3c186

377 files changed

Lines changed: 21519 additions & 10909 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎packages/servicenow/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "2.2.0"
3+
changes:
4+
- description: Add tags to ingest pipeline processors and add `preserve_original_event` to pipeline-level `on_failure` handlers.
5+
type: enhancement
6+
link: https://github.com/elastic/integrations/pull/20574
27
- version: "2.1.1"
38
changes:
49
- description: Set agentless deployment mode `release` field to `ga`.

‎packages/servicenow/data_stream/event/elasticsearch/ingest_pipeline/default.yml‎

Lines changed: 647 additions & 276 deletions
Large diffs are not rendered by default.

‎packages/servicenow/data_stream/event/elasticsearch/ingest_pipeline/pipeline_alm_hardware.yml‎

Lines changed: 75 additions & 27 deletions
Large diffs are not rendered by default.

‎packages/servicenow/data_stream/event/elasticsearch/ingest_pipeline/pipeline_change_request.yml‎

Lines changed: 93 additions & 35 deletions
Large diffs are not rendered by default.

‎packages/servicenow/data_stream/event/elasticsearch/ingest_pipeline/pipeline_cmdb_ci_business_app.yml‎

Lines changed: 47 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -3,51 +3,57 @@ description: Pipeline for processing Business Application logs.
33
processors:
44
- set:
55
field: event.kind
6-
tag: set_event_kind_to_event_cmdb_ci_business_app
6+
tag: set_event_kind_to_event_cmdb_ci_business_app_d452ef2c
77
value: event
88
- set:
99
field: event.category
10-
tag: append_event_category_cmdb_ci_business_app
10+
tag: set_event_category_cmdb_ci_business_app_a46e8e56
1111
value: ['configuration', 'host']
1212
- convert:
1313
field: servicenow.event.active_user_count.value
14-
tag: convert_active_user_count_value_to_long
14+
tag: convert_active_user_count_value_to_long_efacb1c6
1515
type: long
1616
ignore_missing: true
1717
if: ctx.servicenow?.event?.active_user_count?.value != ''
1818
on_failure:
1919
- remove:
20+
tag: remove_servicenow_event_active_user_count_value_c51e6d64
2021
field: servicenow.event.active_user_count.value
2122
- append:
23+
tag: append_error_message_ab766f07
2224
field: error.message
2325
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
2426
- convert:
2527
field: servicenow.event.certified.display_value
26-
tag: convert_certified_display_value_to_boolean
28+
tag: convert_certified_display_value_to_boolean_1bdf6dae
2729
type: boolean
2830
ignore_missing: true
2931
if: ctx.servicenow?.event?.certified?.display_value != ''
3032
on_failure:
3133
- remove:
34+
tag: remove_servicenow_event_certified_display_value_7c82e143
3235
field: servicenow.event.certified.display_value
3336
- append:
37+
tag: append_error_message_66cc41b0
3438
field: error.message
3539
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
3640
- convert:
3741
field: servicenow.event.certified.value
38-
tag: convert_certified_value_to_boolean
42+
tag: convert_certified_value_to_boolean_6d33cafd
3943
type: boolean
4044
ignore_missing: true
4145
if: ctx.servicenow?.event?.certified?.value != ''
4246
on_failure:
4347
- remove:
48+
tag: remove_servicenow_event_certified_value_b9588039
4449
field: servicenow.event.certified.value
4550
- append:
51+
tag: append_error_message_c45904af
4652
field: error.message
4753
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
4854
- date:
4955
field: servicenow.event.contract_end_date.display_value
50-
tag: date_contract_end_date_display_value
56+
tag: date_contract_end_date_display_value_26ad6685
5157
target_field: servicenow.event.contract_end_date.display_value
5258
formats:
5359
- yyyy-MM-dd H:mm:ss
@@ -70,13 +76,15 @@ processors:
7076
if: ctx.servicenow?.event?.contract_end_date?.display_value != null && ctx.servicenow.event.contract_end_date.display_value != '' && ctx._conf?.date_format_preference == "ddMM"
7177
on_failure:
7278
- remove:
79+
tag: remove_servicenow_event_contract_end_date_display_value_92e93a8c
7380
field: servicenow.event.contract_end_date.display_value
7481
- append:
82+
tag: append_error_message_5ce678bb
7583
field: error.message
7684
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
7785
- date:
7886
field: servicenow.event.contract_end_date.display_value
79-
tag: date_contract_end_date_display_value
87+
tag: date_contract_end_date_display_value_d307ee03
8088
target_field: servicenow.event.contract_end_date.display_value
8189
formats:
8290
- yyyy-MM-dd H:mm:ss
@@ -99,13 +107,15 @@ processors:
99107
if: ctx.servicenow?.event?.contract_end_date?.display_value != null && ctx.servicenow.event.contract_end_date.display_value != '' && ctx._conf?.date_format_preference == "MMdd"
100108
on_failure:
101109
- remove:
110+
tag: remove_servicenow_event_contract_end_date_display_value_6b62b016
102111
field: servicenow.event.contract_end_date.display_value
103112
- append:
113+
tag: append_error_message_33869d2e
104114
field: error.message
105115
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
106116
- date:
107117
field: servicenow.event.contract_end_date.value
108-
tag: date_contract_end_date_value
118+
tag: date_contract_end_date_value_d581466b
109119
target_field: servicenow.event.contract_end_date.value
110120
formats:
111121
- yyyy-MM-dd H:mm:ss
@@ -128,13 +138,15 @@ processors:
128138
if: ctx.servicenow?.event?.contract_end_date?.value != null && ctx.servicenow.event.contract_end_date.value != '' && ctx._conf?.date_format_preference == "ddMM"
129139
on_failure:
130140
- remove:
141+
tag: remove_servicenow_event_contract_end_date_value_af166b18
131142
field: servicenow.event.contract_end_date.value
132143
- append:
144+
tag: append_error_message_e541546e
133145
field: error.message
134146
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
135147
- date:
136148
field: servicenow.event.contract_end_date.value
137-
tag: date_contract_end_date_value
149+
tag: date_contract_end_date_value_86755264
138150
target_field: servicenow.event.contract_end_date.value
139151
formats:
140152
- yyyy-MM-dd H:mm:ss
@@ -156,13 +168,15 @@ processors:
156168
if: ctx.servicenow?.event?.contract_end_date?.value != null && ctx.servicenow.event.contract_end_date.value != '' && ctx._conf?.date_format_preference == "MMdd"
157169
on_failure:
158170
- remove:
171+
tag: remove_servicenow_event_contract_end_date_value_666f2ef1
159172
field: servicenow.event.contract_end_date.value
160173
- append:
174+
tag: append_error_message_17b9558c
161175
field: error.message
162176
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
163177
- date:
164178
field: servicenow.event.next_assessment_date.display_value
165-
tag: date_next_assessment_date_display_value
179+
tag: date_next_assessment_date_display_value_81e3a094
166180
target_field: servicenow.event.next_assessment_date.display_value
167181
formats:
168182
- yyyy-MM-dd H:mm:ss
@@ -182,17 +196,18 @@ processors:
182196
- dd.MM.yyyy H:mm:ss
183197
- dd.MM.yyyy HH:mm:ss
184198
- dd.MM.yyyy
185-
if: ctx.servicenow?.event?.next_assessment_date?.display_value != null &&
186-
ctx.servicenow.event.next_assessment_date.display_value != '' && ctx._conf?.date_format_preference == "ddMM"
199+
if: ctx.servicenow?.event?.next_assessment_date?.display_value != null && ctx.servicenow.event.next_assessment_date.display_value != '' && ctx._conf?.date_format_preference == "ddMM"
187200
on_failure:
188201
- remove:
202+
tag: remove_servicenow_event_next_assessment_date_display_value_284b83a9
189203
field: servicenow.event.next_assessment_date.display_value
190204
- append:
205+
tag: append_error_message_f0158a87
191206
field: error.message
192207
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
193208
- date:
194209
field: servicenow.event.next_assessment_date.display_value
195-
tag: date_next_assessment_date_display_value
210+
tag: date_next_assessment_date_display_value_17a71a87
196211
target_field: servicenow.event.next_assessment_date.display_value
197212
formats:
198213
- yyyy-MM-dd H:mm:ss
@@ -212,17 +227,18 @@ processors:
212227
- dd.MM.yyyy H:mm:ss
213228
- dd.MM.yyyy HH:mm:ss
214229
- dd.MM.yyyy
215-
if: ctx.servicenow?.event?.next_assessment_date?.display_value != null &&
216-
ctx.servicenow.event.next_assessment_date.display_value != '' && ctx._conf?.date_format_preference == "MMdd"
230+
if: ctx.servicenow?.event?.next_assessment_date?.display_value != null && ctx.servicenow.event.next_assessment_date.display_value != '' && ctx._conf?.date_format_preference == "MMdd"
217231
on_failure:
218232
- remove:
233+
tag: remove_servicenow_event_next_assessment_date_display_value_4606452b
219234
field: servicenow.event.next_assessment_date.display_value
220235
- append:
236+
tag: append_error_message_c13b8595
221237
field: error.message
222238
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
223239
- date:
224240
field: servicenow.event.next_assessment_date.value
225-
tag: date_next_assessment_date_value
241+
tag: date_next_assessment_date_value_1b31fcc0
226242
target_field: servicenow.event.next_assessment_date.value
227243
formats:
228244
- yyyy-MM-dd H:mm:ss
@@ -245,13 +261,15 @@ processors:
245261
if: ctx.servicenow?.event?.next_assessment_date?.value != null && ctx.servicenow.event.next_assessment_date.value != '' && ctx._conf?.date_format_preference == "ddMM"
246262
on_failure:
247263
- remove:
264+
tag: remove_servicenow_event_next_assessment_date_value_d38d8381
248265
field: servicenow.event.next_assessment_date.value
249266
- append:
267+
tag: append_error_message_437ccaf7
250268
field: error.message
251269
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
252270
- date:
253271
field: servicenow.event.next_assessment_date.value
254-
tag: date_next_assessment_date_value
272+
tag: date_next_assessment_date_value_917f5d12
255273
target_field: servicenow.event.next_assessment_date.value
256274
formats:
257275
- yyyy-MM-dd H:mm:ss
@@ -273,24 +291,29 @@ processors:
273291
if: ctx.servicenow?.event?.next_assessment_date?.value != null && ctx.servicenow.event.next_assessment_date.value != '' && ctx._conf?.date_format_preference == "MMdd"
274292
on_failure:
275293
- remove:
294+
tag: remove_servicenow_event_next_assessment_date_value_1770cc89
276295
field: servicenow.event.next_assessment_date.value
277296
- append:
297+
tag: append_error_message_f19be0cc
278298
field: error.message
279299
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
280300
- convert:
281301
field: servicenow.event.organization_unit_count.value
282-
tag: convert_organization_unit_count_value_to_long
302+
tag: convert_organization_unit_count_value_to_long_f273fed2
283303
type: long
284304
ignore_missing: true
285305
if: ctx.servicenow?.event?.organization_unit_count?.value != ''
286306
on_failure:
287307
- remove:
308+
tag: remove_servicenow_event_organization_unit_count_value_f38decce
288309
field: servicenow.event.organization_unit_count.value
289310
- append:
311+
tag: append_error_message_6b144607
290312
field: error.message
291313
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
292314
on_failure:
293315
- append:
316+
tag: append_error_message_d916120d
294317
field: error.message
295318
value: >-
296319
Processor '{{{ _ingest.on_failure_processor_type }}}'
@@ -300,5 +323,10 @@ on_failure:
300323
}}}'
301324
- set:
302325
field: event.kind
303-
tag: set_pipeline_error_to_event_kind
326+
tag: set_pipeline_error_to_event_kind_72b1902f
304327
value: pipeline_error
328+
- append:
329+
tag: append_tags_279d5a5c
330+
field: tags
331+
value: preserve_original_event
332+
allow_duplicates: false

0 commit comments

Comments
 (0)