SSI: Standardize processor tagging and event.original preservation across owned pipelines #20558

Context

Diagnosing ingest-pipeline failures from telemetry depends on two conventions being applied consistently: every processor carrying a tag, and event.original being preserved on failure. A scan of the 191 packages owned by elastic/security-service-integrations shows both are applied unevenly. This issue tracks bringing the whole set up to standard; it generalizes the GA-candidate work in #20557.

1. Processor tagging

Failure telemetry identifies a processor as type:tag, or a bare type when untagged. Untagged same-type processors collapse into one bucket, so failures cannot be attributed to a specific step.

Current state across 191 packages: 69% of processors tagged in aggregate.

• 6 packages with zero tagged processors: carbonblack_edr, cylance, jamf_compliance_reporter, jamf_protect, lumos, lyve_cloud
• 80 packages below 50%. Worst offenders (≤5%): infoblox_bloxone_ddi (1%), sophos_central (1%), atlassian_confluence (2%), carbon_black_cloud (2%), darktrace (2%), mattermost (2%), santa (2%), tenable_sc (2%), thycotic_ss (2%), ti_eclecticiq (2%), ti_mandiant_advantage (2%), 1password (3%), bbot (3%), lastpass (3%), proofpoint_tap (3%), zerofox (3%), zscaler_zpa (3%), … (full list in the scan output).

Target: every failure-capable processor carries a unique descriptive tag.

2. event.original preservation on failure

Of 797 pipelines with an on_failure block, roughly a quarter do not reference preserve_original_event, meaning a failure may not retain the raw payload for diagnosis. Ordering bugs also exist where event.original is removed before the failure point (see bitsight in ).

Target: every pipeline's on_failure sets event.kind: pipeline_error, appends preserve_original_event to tags, and appends processor context to error.message; and event.original is populated before, and not removed ahead of, any processor that can fail.

Suggested approach

• Prioritize the agentless GA candidates (Ingest-pipeline diagnosability fixes required for the 19 beta→GA promotions #20557) and the zero-tag / sub-5% packages first.
• Automation exists that can add event.original preservation; worth reusing here.
• elastic-package can add tags using: elastic-package modify -m pipeline-tag
• Consider a lint/CI check so new pipelines can't regress (untagged processors, missing preserve_original_event in on_failure).

Measurement

Numbers above are from parsing top-level processors in packages/*/data_stream/*/elasticsearch/ingest_pipeline/*.yml on main. Percentages exclude processors nested inside foreach/on_failure, and the event.original figure is a heuristic — treat per-package status as a starting point to verify, not a certification.