Repository navigation
Retain event.original value upon pipeline errors #12045
Description
When an ingest pipeline error occurs, our log pipelines should retain the event.original value to ensure that no data loss occurs and to facilitate correcting the failure. If processing is interrupted due to an error, some data may not have been extracted (i.e. incomplete processing) so it's important to retain the event.original. And secondly, in order for the package maintainers to be able to take action on reports of pipeline failures they nearly always need the event.original value to reproduce and understand the issue.
To implement this we should complete the work related to #10072. This not strictly required, but it helps ensure the event.original is consistently handled. We want the Fleet final_pipeline to be responsible for deleting event.original when tags does not contain preserve_original_event.
Next, in the primary pipeline of each log data stream we update the global on_failure handler to inject preserve_original_event into tags. This will accompany event.kind: pipeline_error. With this mechanism, users can still override this behavior through the various levels of @custom pipelines by deleting the tag value. I expect the work to be accomplished "mechanically", and this mechanism can be applied separately on the integrations owned by each SIT team.
TODO
- Tag events with
preserve_original_eventin primary on_failure handlers
Activity
elasticmachine commentedon Dec 9, 2024
Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)
elasticmachine commentedon Dec 9, 2024
Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)
elasticmachine commentedon Dec 9, 2024
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
elasticmachine commentedon Dec 9, 2024
Pinging @elastic/security-scalability (Team:Security-Scalability)
elasticmachine commentedon Dec 9, 2024
Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)
andrewkroh commentedon Sep 17, 2025
#12046 contains the script that was used to apply changes to the security service team integrations.