Skip to content

add processor tags and preserve_original_event on failure (1/4) #20572

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kcreddy merged 8 commits into from
Aug 20, 2026
+23,845 −12,509
Merged

add processor tags and preserve_original_event on failure (1/4) #20572

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
48d46df
ssi: add processor tags and preserve_original_event on failure (1/4)
kcreddy Aug 6, 2026
3c706fe
ssi: update changelog links to PR #20572
kcreddy Aug 6, 2026
e741147
Merge remote-tracking branch 'upstream/main' into tagging-evt-origina…
kcreddy Aug 13, 2026
7cd7b13
ssi: add processor tags and preserve_original_event on failure (1/4)
kcreddy Aug 13, 2026
7ae0c0b
fix tags with asterisk inside the tags
kcreddy Aug 18, 2026
b714aa1
ssi: address review feedback on PR 20572 (1/4)
kcreddy Aug 19, 2026
bb93b22
Merge remote-tracking branch 'upstream/main' into tagging-evt-origina…
kcreddy Aug 19, 2026
34077ab
Merge remote-tracking branch 'upstream/main' into tagging-evt-origina…
kcreddy Aug 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
5 changes: 5 additions & 0 deletions packages/1password/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top

Check notice on line 1 in packages/1password/changelog.yml

View workflow job for this annotation

GitHub Actions / Lint user-facing content 

Elastic.Versions: Use 'later versions' instead of 'newer versions' when referring to versions.
- version: "1.38.0"
changes:
- description: Add tags to ingest pipeline processors.
type: enhancement
link: https://github.com/elastic/integrations/pull/20572
- version: "1.37.0"
changes:
- description: Set 'elastic' owner type.
Expand Down
33 changes: 32 additions & 1 deletion packages/1password/data_stream/audit_events/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,83 +8,100 @@ processors:
- team
ignore_missing: true
if: ctx.organization instanceof String && ctx.division instanceof String && ctx.team instanceof String
tag: remove_agentless_tags
tag: remove_agentless_tags_e6856012
description: >-
Removes the fields added by Agentless as metadata,
as they can collide with ECS fields.
- rename:
tag: rename_message_to_event_original_56a77271
field: message
target_field: event.original
ignore_missing: true
if: ctx.event?.original == null
- remove:
tag: remove_message_2ee3a7d4
field: message
ignore_missing: true
if: ctx.event?.original != null
description: 'The `message` field is no longer required if the document has an `event.original` field.'
- json:
tag: json_event_original_to_onepassword_da036177
field: event.original
target_field: onepassword
- drop:
tag: drop_no_timestamp_invalid_json_02acf80d
description: Drop if no timestamp (invalid json)
if: "ctx?.onepassword?.timestamp == null"
#######################
## ECS Event Mapping ##
#######################
- set:
tag: set_ecs_version_0f7334bb
field: ecs.version
value: "8.11.0"
# Sets event.created from the @timestamp field generated by filebeat before being overwritten further down
- set:
tag: set_event_created_e3f09e3b
field: event.created
copy_from: "@timestamp"
- set:
tag: set_event_kind_de80643c
field: event.kind
value: event
- append:
tag: append_event_category_c8ab9fa2
field: event.category
value: [configuration]
- append:
tag: append_event_type_8860cef4
field: event.type
value: [access]
- rename:
tag: rename_onepassword_action_to_event_action_24171185
field: onepassword.action
target_field: event.action
ignore_missing: true
#########################
## ECS Related Mapping ##
#########################
- append:
tag: append_related_user_572fd854
field: related.user
value: "{{{onepassword.actor_details.name}}}"
allow_duplicates: false
if: ctx.onepassword?.actor_details?.name != null
- append:
tag: append_related_user_6d2a829f
field: related.user
value: "{{{onepassword.actor_uuid}}}"
allow_duplicates: false
if: ctx.onepassword?.actor_uuid != null
- append:
tag: append_related_user_ec2b4068
field: related.user
value: "{{{onepassword.object_uuid}}}"
allow_duplicates: false
if: ctx.onepassword?.object_type == "user" && ctx?.onepassword?.object_uuid != null
- append:
tag: append_related_user_54a15ffd
field: related.user
value: "{{{onepassword.aux_uuid}}}"
allow_duplicates: false
if: ctx.onepassword?.object_type == "gm" && ctx?.onepassword?.aux_uuid != null
- append:
tag: append_related_user_b2338d93
field: related.user
value: "{{{onepassword.aux_uuid}}}"
allow_duplicates: false
if: ctx.onepassword?.object_type == "device" && ctx?.onepassword?.aux_uuid != null
- append:
tag: append_related_user_f39de967
field: related.user
value: "{{{onepassword.aux_uuid}}}"
allow_duplicates: false
if: ctx.onepassword?.object_type == "uva" && ctx?.onepassword?.aux_uuid != null
- append:
tag: append_related_ip_93983512
field: related.ip
value: "{{{onepassword.session.ip}}}"
allow_duplicates: false
Expand All @@ -93,32 +110,39 @@ processors:
## ECS User Mapping ##
######################
- rename:
tag: rename_onepassword_actor_uuid_to_user_id_60afb126
field: onepassword.actor_uuid
target_field: user.id
ignore_missing: true
- rename:
tag: rename_onepassword_actor_details_name_to_user_full_name_c880bb1c
field: onepassword.actor_details.name
target_field: user.full_name
ignore_missing: true
- set:
tag: set_user_name_67fd4e49
field: user.name
copy_from: user.full_name
ignore_empty_value: true
- rename:
tag: rename_onepassword_actor_details_email_to_user_email_1e9bf80a
field: onepassword.actor_details.email
target_field: user.email
ignore_missing: true
########################
## ECS Source Mapping ##
########################
- rename:
tag: rename_onepassword_session_ip_to_source_ip_23f74ca6
field: onepassword.session.ip
target_field: source.ip
ignore_missing: true
- geoip:
tag: geoip_source_ip_to_source_geo_0494c6b1
field: source.ip
target_field: source.geo
- geoip:
tag: geoip_source_ip_to_source_as_28d69883
database_file: GeoLite2-ASN.mmdb
field: source.ip
target_field: source.as
Expand All @@ -127,37 +151,44 @@ processors:
- organization_name
ignore_missing: true
- rename:
tag: rename_source_as_asn_to_source_as_number_a917047d
field: source.as.asn
target_field: source.as.number
ignore_missing: true
- rename:
tag: rename_source_as_organization_name_to_source_as_organization_name_f1362d0b
field: source.as.organization_name
target_field: source.as.organization.name
ignore_missing: true
######################
## ECS Base Mapping ##
######################
- date:
tag: date_onepassword_timestamp_bea6075a
field: onepassword.timestamp
formats:
- ISO8601
#############
## Cleanup ##
#############
- remove:
tag: remove_cleanup_49b6d979
field:
- onepassword.timestamp
- onepassword.user
- onepassword.location # Use the included GeoIP processor
ignore_missing: true
on_failure:
- set:
tag: set_event_kind_72b1902f
field: event.kind
value: pipeline_error
- append:
tag: append_tags_279d5a5c
field: tags
value: preserve_original_event
allow_duplicates: false
- append:
tag: append_error_message_206bb8ac
field: error.message
value: '{{{ _ingest.on_failure_message }}}'
32 changes: 31 additions & 1 deletion packages/1password/data_stream/item_usages/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,68 +8,82 @@ processors:
- team
ignore_missing: true
if: ctx.organization instanceof String && ctx.division instanceof String && ctx.team instanceof String
tag: remove_agentless_tags
tag: remove_agentless_tags_e6856012
description: >-
Removes the fields added by Agentless as metadata,
as they can collide with ECS fields.
- rename:
tag: rename_message_to_event_original_56a77271
field: message
target_field: event.original
ignore_missing: true
if: ctx.event?.original == null
- remove:
tag: remove_message_2ee3a7d4
field: message
ignore_missing: true
if: ctx.event?.original != null
description: 'The `message` field is no longer required if the document has an `event.original` field.'
- json:
tag: json_event_original_to_onepassword_da036177
field: event.original
target_field: onepassword
- drop:
tag: drop_no_timestamp_invalid_json_02acf80d
description: Drop if no timestamp (invalid json)
if: "ctx?.onepassword?.timestamp == null"
#######################
## ECS Event Mapping ##
#######################
- set:
tag: set_ecs_version_0f7334bb
field: ecs.version
value: '8.11.0'
# Sets event.created from the @timestamp field generated by filebeat before being overwritten further down
- set:
tag: set_event_created_e3f09e3b
field: event.created
copy_from: "@timestamp"
- set:
tag: set_event_kind_de80643c
field: event.kind
value: event
- append:
tag: append_event_category_a495f7f8
field: event.category
value: [file]
- append:
tag: append_event_type_8860cef4
field: event.type
value: [access]
- rename:
tag: rename_onepassword_action_to_event_action_24171185
field: onepassword.action
target_field: event.action
ignore_missing: true
#########################
## ECS Related Mapping ##
#########################
- append:
tag: append_related_user_6bd9ebd3
field: related.user
value: "{{{onepassword.user.uuid}}}"
allow_duplicates: false
if: ctx?.onepassword?.user?.uuid != null
- append:
tag: append_related_user_59c6eae3
field: related.user
value: "{{{onepassword.user.email}}}"
allow_duplicates: false
if: ctx?.onepassword?.user?.email != null
- append:
tag: append_related_user_03367e37
field: related.user
value: "{{{onepassword.user.name}}}"
allow_duplicates: false
if: ctx?.onepassword?.user?.name != null
- append:
tag: append_related_ip_9c1d4a47
field: related.ip
value: "{{{onepassword.client.ip_address}}}"
allow_duplicates: false
Expand All @@ -78,43 +92,52 @@ processors:
## ECS User Mapping ##
######################
- rename:
tag: rename_onepassword_user_uuid_to_user_id_e1ae97c7
field: onepassword.user.uuid
target_field: user.id
ignore_missing: true
- rename:
tag: rename_onepassword_user_name_to_user_full_name_14ca796f
field: onepassword.user.name
target_field: user.full_name
ignore_missing: true
- set:
tag: set_user_name_67fd4e49
field: user.name
copy_from: user.full_name
ignore_empty_value: true
- rename:
tag: rename_onepassword_user_email_to_user_email_de5b00df
field: onepassword.user.email
target_field: user.email
ignore_missing: true
####################
## ECS OS Mapping ##
####################
- rename:
tag: rename_onepassword_client_os_name_to_host_os_name_700b63f5
field: onepassword.client.os_name
target_field: host.os.name
ignore_missing: true
- rename:
tag: rename_onepassword_client_os_version_to_host_os_version_245989d7
field: onepassword.client.os_version
target_field: host.os.version
ignore_missing: true
########################
## ECS Source Mapping ##
########################
- rename:
tag: rename_onepassword_client_ip_address_to_source_ip_40e01f74
field: onepassword.client.ip_address
target_field: source.ip
ignore_missing: true
- geoip:
tag: geoip_source_ip_to_source_geo_0494c6b1
field: source.ip
target_field: source.geo
- geoip:
tag: geoip_source_ip_to_source_as_28d69883
database_file: GeoLite2-ASN.mmdb
field: source.ip
target_field: source.as
Expand All @@ -123,37 +146,44 @@ processors:
- organization_name
ignore_missing: true
- rename:
tag: rename_source_as_asn_to_source_as_number_a917047d
field: source.as.asn
target_field: source.as.number
ignore_missing: true
- rename:
tag: rename_source_as_organization_name_to_source_as_organization_name_f1362d0b
field: source.as.organization_name
target_field: source.as.organization.name
ignore_missing: true
######################
## ECS Base Mapping ##
######################
- date:
tag: date_onepassword_timestamp_bea6075a
field: onepassword.timestamp
formats:
- ISO8601
#############
## Cleanup ##
#############
- remove:
tag: remove_cleanup_49b6d979
field:
- onepassword.timestamp
- onepassword.user
- onepassword.location # Use the included GeoIP processor
ignore_missing: true
on_failure:
- set:
tag: set_event_kind_72b1902f
field: event.kind
value: pipeline_error
- append:
tag: append_tags_279d5a5c
field: tags
value: preserve_original_event
allow_duplicates: false
- append:
tag: append_error_message_206bb8ac
field: error.message
value: '{{{ _ingest.on_failure_message }}}'
Loading
Loading