Remove event.original removal processors from ingest pipelines

@Custom

It seems that most of the checked integrations have a removal processor for event.original (except if it contains the preserve tag) in their pipelines.
In the other way, the .fleet-final-pipeline also has this processor and runs after all integration pipelines (incl. the @Custom pipelines) have finished. If the integration has this removal, the @Custom pipeline cannot parse the original event anymore as it is not present in the event anymore.

I would suggest removing the removal processors from all integration ingest pipelines and forward the removal functionality into the fleet-final-pipeline where it belongs. In that case @Custom pipelines may parse the original event and can extract additional data or correct malformed parsing in the generic integrations (e.g. catalina.out in the Apache Tomcat integration):

I think this is a good idea. I will occasional install a logs@custom pipeline that keeps the event.original when there is a pipeline failure so that I can debug it.

PUT _ingest/pipeline/logs@custom
{
  "processors": [
    {
      "append": {
        "field": "tags",
        "value": [
          "preserve_original_event"
        ],
        "allow_duplicates": false,
        "tag": "ctx.error?.message != null",
        "ignore_failure": true
      }
    }
  ]
}

And in order for this to work, I currently have to manually drop that remove processor from the managed pipelines.

If that change would be approved, I maybe start working on this after my current PR has been merged.

Remove event.original removal processors (integrations 1Password to A…

Remove event.original removal processors (integrations B* to C*) (#10897

Remove event.original removal processors (integrations aws to azure) (#…

Relates #7636

This work was partially completed through several pull requests. To finish this out, I think we should script the changes. We can review edits to each team's integrations separately to optimize the process. Basically the script needs to delete the remove processor for the event.original field and ensure that the integration requires at least >=8.11.0 (where the Fleet final pipeline changes were introduced).

Remove event.original removal processors (integrations 1Password to A…

Remove event.original removal processors (integrations B* to C*) (ela…

Remove event.original removal processors (integrations aws to azure) (e…

Remove event.original removal processors (integrations 1Password to A…

Remove event.original removal processors (integrations B* to C*) (ela…

Remove event.original removal processors (integrations aws to azure) (e…

Opened additional PR's to finally fulfil this issue. Please review accordingly

Removed event.original processors from network device integrations (#…

@taylor-swanson

@taylor-swanson This issue is not complete until the following PR has been merged: #13518

@Alphayeeeet

@Alphayeeeet, this issue was auto-closed by github in #13518. I'll reopen it.

Be mindful of using the word close next to the issue in the PR description, as github will interpret this as it should close the issue when the PR merges.

I usually write Related: #10072 to avoid this situation if the parent issue has a number of sub-issues/PRs under it.

@taylor-swanson

@taylor-swanson Thank you for mentioning that. I must admit, I wasn't aware of that. I will keep it in mind for future PR's.

Navigation Menu

Repository navigation

Remove event.original removal processors from ingest pipelines #10072

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Remove event.original removal processors from ingest pipelines #10072

Description

Activity

andrewkroh commented on Jun 6, 2024

Alphayeeeet commented on Jun 20, 2024

andrewkroh commented on Dec 9, 2024

andrewkroh commented on Dec 9, 2024

Alphayeeeet commented on Apr 12, 2025

Alphayeeeet commented on Jul 3, 2025

taylor-swanson commented on Jul 7, 2025

Alphayeeeet commented on Jul 7, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions