Repository navigation
Remove event.original removal processors from ingest pipelines #10072
Description
Activity
I think this is a good idea. I will occasional install a
logs@custompipeline that keeps theevent.originalwhen there is a pipeline failure so that I can debug it.PUT _ingest/pipeline/logs@custom { "processors": [ { "append": { "field": "tags", "value": [ "preserve_original_event" ], "allow_duplicates": false, "tag": "ctx.error?.message != null", "ignore_failure": true } } ] }And in order for this to work, I currently have to manually drop that
removeprocessor from the managed pipelines.If that change would be approved, I maybe start working on this after my current PR has been merged.
- added a commit that references this issue
on Aug 13, 2024 - added a commit that references this issue
on Sep 3, 2024 - added a commit that references this issue
on Sep 23, 2024 Relates #7636
This work was partially completed through several pull requests. To finish this out, I think we should script the changes. We can review edits to each team's integrations separately to optimize the process. Basically the script needs to delete the
removeprocessor for theevent.originalfield and ensure that the integration requires at least >=8.11.0 (where the Fleet final pipeline changes were introduced).- added 6 commits that reference this issue
on Feb 4, 2025 Opened additional PR's to finally fulfil this issue. Please review accordingly
- added a commit that references this issue
on Jun 13, 2025 @taylor-swanson This issue is not complete until the following PR has been merged: #13518
@Alphayeeeet, this issue was auto-closed by github in #13518. I'll reopen it.
Be mindful of using the word
closenext to the issue in the PR description, as github will interpret this as it should close the issue when the PR merges.
I usually write
Related: #10072to avoid this situation if the parent issue has a number of sub-issues/PRs under it.@taylor-swanson Thank you for mentioning that. I must admit, I wasn't aware of that. I will keep it in mind for future PR's.
- added a parent issue
on Jul 8, 2025
It seems that most of the checked integrations have a removal processor for event.original (except if it contains the preserve tag) in their pipelines.
In the other way, the .fleet-final-pipeline also has this processor and runs after all integration pipelines (incl. the @Custom pipelines) have finished. If the integration has this removal, the @Custom pipeline cannot parse the original event anymore as it is not present in the event anymore.
I would suggest removing the removal processors from all integration ingest pipelines and forward the removal functionality into the fleet-final-pipeline where it belongs. In that case @Custom pipelines may parse the original event and can extract additional data or correct malformed parsing in the generic integrations (e.g. catalina.out in the Apache Tomcat integration):