Skip to content

[auditd] Field schema normalization: align auditd and auditd_manager namespaces #19080

Open
Open
[auditd] Field schema normalization: align auditd and auditd_manager namespaces#19080
Assignees
efd6
Labels
9.5 candidateIntegration:auditdAuditd LogsIntegration:auditd_managerAuditd ManagerTeam: XDRTeam:Security-Service IntegrationsSecurity Service Integrations team [elastic/security-service-integrations]enhancementNew feature or request

Description

@ricardoamaro
ricardoamaro
opened on May 19, 2026

Problem

The auditd (Auditd Logs) and auditd_manager (Auditd Manager) integrations produce incompatible field namespaces:

  • auditd → auditd.log.* (grok/regex ingest pipeline, ~100 fields)
  • auditd_manager → auditd.data.* (go-libaudit agent-side parsing, 300+ fields)

Because field names and values differ between the two, detection rules cannot target both integrations without duplicating every field condition. In practice, the detection-rules team has standardized entirely on auditd_manager: approximately 160 rules reference this integration, zero reference auditd directly. Customers who cannot install auditd_manager (managed environments, restricted installs) get no rule coverage.

Current state (as of May 2026)

Phase 1 work is underway:

The false default ensures no breakage for users on older Filebeat versions that don't have the new parser. Users opt in, validate, then the default flips.

Goal

Once Phase 1 is stable, evaluate normalizing the auditd integration's field output to align with the auditd_manager namespace (auditd.data.*), so that detection rules can target both integrations with a single field reference.

Work needed

  • Phase 1: Land beats#50791 + integrations#19071 and validate in production
  • Phase 2: Map auparse output fields to auditd.data.* namespace (or agree on a shared ECS-aligned namespace)
  • Phase 2: Define deprecation/migration path for existing auditd.log.* users (custom rules, dashboards)
  • Phase 2: Update integration manifest with minimum Filebeat version requirement
  • Cross-team: Coordinate with detection-rules team to review ~160 rules once schema is normalized; classify each as auditd-only, auditd_manager-only, or both

Stakeholders

  • elastic/sec-linux-platform: owns both integration packages (CODEOWNERS), driving Phase 1 and 2
  • Detection-rules team: consumer; ~160 rules need review post-normalization

Related

Activity

  1. ricardoamaro
    assigned
    Tacklebox
    on May 19, 2026
  2. ricardoamaro
    added
    enhancementNew feature or request
    Integration:auditdAuditd Logs
    Integration:auditd_managerAuditd Manager
    Team:Security-Linux PlatformLinux Platform Security team [elastic/sec-linux-platform]
    on May 19, 2026

  3. infra-vault-gh-plugin-prod commented on May 19, 2026

    @infra-vault-gh-plugin-prod
    infra-vault-gh-plugin-prodBot
    on May 19, 2026

    Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

  4. ricardoamaro
    unassigned
    Tacklebox
    on May 19, 2026

  5. github-actions commented on May 19, 2026

    @github-actions
    github-actionsBot
    on May 19, 2026 – with GitHub Actions
    Contributor

    tl;dr: auditd and auditd_manager currently use different payload roots (auditd.log.* vs auditd.data.*), and a direct rename would break existing saved searches/detections; a phased compatibility-first normalization is the safest path.

    Recommendation

    Adopt a phased normalization plan:

    1. Choose one canonical namespace for new content (recommend auditd.data.* for cross-package alignment).
    2. Add compatibility mappings in pipelines (dual-write/copy for shared core fields) instead of immediate renames.
    3. Migrate dashboards/rules/docs in phases, then deprecate legacy paths in a major version.

    Findings

    Evidence from code and docs
    • auditd package is rooted at auditd.log:
      • packages/auditd/data_stream/log/fields/fields.yml:1
      • packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml:19 (and many other auditd.log.* references)
      • packages/auditd/docs/README.md:25-33 (example event shows auditd.log)
    • auditd_manager package is rooted at auditd.data:
      • packages/auditd_manager/data_stream/auditd/fields/fields.yml:61 (auditd.data.reset) and many auditd.data.* fields
      • packages/auditd_manager/data_stream/auditd/elasticsearch/ingest_pipeline/default.yml:100 (auditd.data.id) and additional auditd.data.* processors
      • packages/auditd_manager/docs/README.md:162-184 (example event shows auditd.data)
    • Both packages identify as the same module (auditd), so divergence is at field schema level:
      • packages/auditd/data_stream/log/fields/base-fields.yml:10-17
      • packages/auditd_manager/data_stream/auditd/fields/base-fields.yml:10-17
    • Downstream content already depends on both namespaces:
      • packages/auditd/kibana/search/auditd-4ac0a370-0a11-11e7-8b04-eb22a5669f27.json:5 (auditd.log.sequence)
      • packages/auditd_manager/kibana/search/auditd_manager-5438b030-c246-11e7-8692-232bd1143e8a.json:10 (auditd.data.exit)
      • packages/security_detection_engine/kibana/security_rule/32300431-c2d5-432d-8ec8-0e03f9924756_9.json:18,41 (auditd.data.syscall)
    • Prior package history also explicitly references auditd.data.* updates:
      • packages/auditd_manager/changelog.yml:19-21 ("Updated field definitions for auditd.data.* fields")

    Verification

    Commands and output 
    $ grep -n '^- name: auditd\.' packages/auditd/data_stream/log/fields/fields.yml | head -30
1:- name: auditd.log

$ grep -n '^- name: auditd\.' packages/auditd_manager/data_stream/auditd/fields/fields.yml | head -30
...
61:- name: auditd.data.reset
69:- name: auditd.messages
...

$ grep -o 'auditd\.log\.' packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml | wc -l
94

$ grep -o 'auditd\.data\.' packages/auditd_manager/data_stream/auditd/elasticsearch/ingest_pipeline/default.yml | wc -l
21

    Detailed Action Plan

    Implementation plan
    1. Define canonical namespace and migration policy in package docs/changelog (both packages).
    2. In packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml, add compatibility writes from high-value auditd.log.* fields to auditd.data.* equivalents (keep auditd.log.* unchanged for now).
    3. Add corresponding field declarations for compatibility fields in packages/auditd/data_stream/log/fields/fields.yml.
    4. Keep auditd_manager namespace stable; only add compatibility aliases where needed for cross-package UX parity.
    5. Update saved searches/dashboards/rules to canonical paths incrementally:
      • packages/auditd/kibana/**
      • packages/auditd_manager/kibana/**
      • packages/security_detection_engine/kibana/security_rule/**
    6. Expand pipeline expected-event fixtures in both packages to validate compatibility fields during transition:
      • packages/auditd/data_stream/log/_dev/test/pipeline/*-expected.json
      • packages/auditd_manager/data_stream/auditd/_dev/test/pipeline/*-expected.json
    7. Announce deprecation window and remove legacy namespace only in a major package version.

    Related Items

    Issues, PRs, files
    Type Link / Path Relevance
    PR #11336 Recent auditd_manager work; not a namespace migration itself
    File packages/auditd/data_stream/log/fields/fields.yml:1 auditd.log root in auditd
    File packages/auditd_manager/data_stream/auditd/fields/fields.yml:61 auditd.data root fields in auditd_manager
    File packages/security_detection_engine/kibana/security_rule/32300431-c2d5-432d-8ec8-0e03f9924756_9.json:18 Detection content uses auditd.data.*
    Issue #6525, #4125 Found by search but content unavailable in this run due integrity filtering

    Note

    🔒 Integrity filter blocked 19 items

    The following items were blocked because they don't meet the GitHub integrity level.

    To allow these resources, lower min-integrity in your GitHub frontmatter:

    tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

    What is this? | From workflow: Issue Triage

    Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

  6. ricardoamaro commented on May 20, 2026

    @ricardoamaro
    ricardoamaro
    on May 20, 2026
    ContributorAuthor

    @Aegrah @nfritts for awareness

  7. 111andre111 commented on May 20, 2026

    @111andre111
    111andre111
    on May 20, 2026

    cc @Mikaayenson @approksiu
    Once that work is done, we could review the auditd rules if these could relate as well partially to the Auditd integration.

  8. ricardoamaro
    self-assigned this
    on May 20, 2026
  9. ricardoamaro
    added
    Epic
    on Jun 3, 2026
  10. ricardoamaro
    added
    9.5 candidate
    Team: XDR
    on Jun 17, 2026
  11. jamiehynds
    removed
    Epic
    on Jun 18, 2026

  12. approksiu commented on Jul 2, 2026

    @approksiu
    approksiu
    on Jul 2, 2026
    Contributor

    @jamiehynds is this still in the works? Do you know when this work will be released? Asking because prebuilt detection rules have a dependency on this. Thank you!

  13. jamiehynds
    added
    Team:Security-Service IntegrationsSecurity Service Integrations team [elastic/security-service-integrations]
    on Jul 7, 2026

  14. infra-vault-gh-plugin-prod commented on Jul 7, 2026

    @infra-vault-gh-plugin-prod
    infra-vault-gh-plugin-prodBot
    on Jul 7, 2026

    Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

  15. Aegrah commented on Jul 27, 2026

    @Aegrah
    Aegrah
    on Jul 27, 2026

    Hey, do we have any progress update on this one? We have several SDH's related to this feature! Much appreciated.

  16. narph
    assigned
    efd6
    and unassigned
    ricardoamaro
    on Jul 28, 2026

  17. jamiehynds commented on Jul 29, 2026

    @jamiehynds
    jamiehynds
    on Jul 29, 2026

    @Aegrah @approksiu - @efd6 has now been assigned to this issue and will review current state and discuss a path forward.

    My main concern here is the nature of the breaking change in phase 2 - we have a large install base using the auditd integration and if we switch from auditd.log to auditd.data, it will break their custom created rules, dashboards, etc. Some discussion needed on a path forward without negatively impacting those users.

  18. narph
    removed
    Team:Security-Linux PlatformLinux Platform Security team [elastic/sec-linux-platform]
    on Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

9.5 candidateIntegration:auditdAuditd LogsIntegration:auditd_managerAuditd ManagerTeam: XDRTeam:Security-Service IntegrationsSecurity Service Integrations team [elastic/security-service-integrations]enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions