Repository navigation
[auditd] Field schema normalization: align auditd and auditd_manager namespaces #19080
Description
Activity
Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)
tl;dr:
auditdandauditd_managercurrently use different payload roots (auditd.log.*vsauditd.data.*), and a direct rename would break existing saved searches/detections; a phased compatibility-first normalization is the safest path.Recommendation
Adopt a phased normalization plan:
- Choose one canonical namespace for new content (recommend
auditd.data.*for cross-package alignment). - Add compatibility mappings in pipelines (dual-write/copy for shared core fields) instead of immediate renames.
- Migrate dashboards/rules/docs in phases, then deprecate legacy paths in a major version.
Findings
Evidence from code and docs
auditdpackage is rooted atauditd.log:packages/auditd/data_stream/log/fields/fields.yml:1packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml:19(and many otherauditd.log.*references)packages/auditd/docs/README.md:25-33(example event showsauditd.log)
auditd_managerpackage is rooted atauditd.data:packages/auditd_manager/data_stream/auditd/fields/fields.yml:61(auditd.data.reset) and manyauditd.data.*fieldspackages/auditd_manager/data_stream/auditd/elasticsearch/ingest_pipeline/default.yml:100(auditd.data.id) and additionalauditd.data.*processorspackages/auditd_manager/docs/README.md:162-184(example event showsauditd.data)
- Both packages identify as the same module (
auditd), so divergence is at field schema level:packages/auditd/data_stream/log/fields/base-fields.yml:10-17packages/auditd_manager/data_stream/auditd/fields/base-fields.yml:10-17
- Downstream content already depends on both namespaces:
packages/auditd/kibana/search/auditd-4ac0a370-0a11-11e7-8b04-eb22a5669f27.json:5(auditd.log.sequence)packages/auditd_manager/kibana/search/auditd_manager-5438b030-c246-11e7-8692-232bd1143e8a.json:10(auditd.data.exit)packages/security_detection_engine/kibana/security_rule/32300431-c2d5-432d-8ec8-0e03f9924756_9.json:18,41(auditd.data.syscall)
- Prior package history also explicitly references
auditd.data.*updates:packages/auditd_manager/changelog.yml:19-21("Updated field definitions forauditd.data.*fields")
Verification
Commands and output
$ grep -n '^- name: auditd\.' packages/auditd/data_stream/log/fields/fields.yml | head -30 1:- name: auditd.log $ grep -n '^- name: auditd\.' packages/auditd_manager/data_stream/auditd/fields/fields.yml | head -30 ... 61:- name: auditd.data.reset 69:- name: auditd.messages ... $ grep -o 'auditd\.log\.' packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml | wc -l 94 $ grep -o 'auditd\.data\.' packages/auditd_manager/data_stream/auditd/elasticsearch/ingest_pipeline/default.yml | wc -l 21
Detailed Action Plan
Implementation plan
- Define canonical namespace and migration policy in package docs/changelog (both packages).
- In
packages/auditd/data_stream/log/elasticsearch/ingest_pipeline/default.yml, add compatibility writes from high-valueauditd.log.*fields toauditd.data.*equivalents (keepauditd.log.*unchanged for now). - Add corresponding field declarations for compatibility fields in
packages/auditd/data_stream/log/fields/fields.yml. - Keep
auditd_managernamespace stable; only add compatibility aliases where needed for cross-package UX parity. - Update saved searches/dashboards/rules to canonical paths incrementally:
packages/auditd/kibana/**packages/auditd_manager/kibana/**packages/security_detection_engine/kibana/security_rule/**
- Expand pipeline expected-event fixtures in both packages to validate compatibility fields during transition:
packages/auditd/data_stream/log/_dev/test/pipeline/*-expected.jsonpackages/auditd_manager/data_stream/auditd/_dev/test/pipeline/*-expected.json
- Announce deprecation window and remove legacy namespace only in a major package version.
Related Items
Issues, PRs, files
Type Link / Path Relevance PR #11336 Recent auditd_managerwork; not a namespace migration itselfFile packages/auditd/data_stream/log/fields/fields.yml:1auditd.logroot in auditdFile packages/auditd_manager/data_stream/auditd/fields/fields.yml:61auditd.dataroot fields in auditd_managerFile packages/security_detection_engine/kibana/security_rule/32300431-c2d5-432d-8ec8-0e03f9924756_9.json:18Detection content uses auditd.data.*Issue #6525, #4125 Found by search but content unavailable in this run due integrity filtering Note
🔒 Integrity filter blocked 19 items
The following items were blocked because they don't meet the GitHub integrity level.
- #19080
issue_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - #19080
search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - #11336
search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - #6525
search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - #4125
search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - [auditd] Field schema normalization: align auditd and auditd_manager namespaces #19080
list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - [Github]: ruleset_* audit fields absent from fields.yml, causing non-deterministic ES|QL verification_exception after index rollover #19079
list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - [google_workspace]: Reduce default lag time and request interval for data streams #19072
list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - crowdstrike: dashboard enhancements #19058
list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - [microsoft_exchange_server]: copy fields into
related#19057list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - [Multiple Integrations]:
data_stream.namespacemapping error on transforms #19054list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - [CrowdStrike] Add support for NG-SIEM in alert pipeline #19048
list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - [mimecast]: Pipeline error for avlog events in siem_logs dataset #19032
list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - [LogsDB] [Stack 8.19.16-SNAPSHOT] [aws] Failing test daily: system test: default in aws.redshift #19021
list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - [Stack 8.19.16-SNAPSHOT] [aws] Failing test daily: system test: default in aws.redshift #19020
list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - [Audit]: event.action derived only from USER_CHAUTHTOK type leads to incorrect classification (op field ignored) #19001
list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - ... and 3 more items
To allow these resources, lower
min-integrityin your GitHub frontmatter:tools: github: min-integrity: approved # merged | approved | unapproved | none
What is this? | From workflow: Issue Triage
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
Reacted by Balu- Choose one canonical namespace for new content (recommend
cc @Mikaayenson @approksiu
Once that work is done, we could review the auditd rules if these could relate as well partially to the Auditd integration.Reacted by Mike@jamiehynds is this still in the works? Do you know when this work will be released? Asking because prebuilt detection rules have a dependency on this. Thank you!
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)
Hey, do we have any progress update on this one? We have several SDH's related to this feature! Much appreciated.
@Aegrah @approksiu - @efd6 has now been assigned to this issue and will review current state and discuss a path forward.
My main concern here is the nature of the breaking change in phase 2 - we have a large install base using the auditd integration and if we switch from auditd.log to auditd.data, it will break their custom created rules, dashboards, etc. Some discussion needed on a path forward without negatively impacting those users.
Reacted by Ruben Groenewoud
Problem
The
auditd(Auditd Logs) andauditd_manager(Auditd Manager) integrations produce incompatible field namespaces:auditd→auditd.log.*(grok/regex ingest pipeline, ~100 fields)auditd_manager→auditd.data.*(go-libaudit agent-side parsing, 300+ fields)Because field names and values differ between the two, detection rules cannot target both integrations without duplicating every field condition. In practice, the detection-rules team has standardized entirely on
auditd_manager: approximately 160 rules reference this integration, zero referenceauditddirectly. Customers who cannot installauditd_manager(managed environments, restricted installs) get no rule coverage.Current state (as of May 2026)
Phase 1 work is underway:
auditdfilestream parser tolibbeat/reader/auditdusinggo-libaudit/auparse. Replaces the 2,371-line ingest pipeline (4 groks + Painless) with a single auparse call per line. Currently outputsauditd.log.*fields.use_filebeat_parsertoggle (defaultfalse) to the auditd data stream, guarding grok processors for pre-parsed events. Beats PR must merge first.The
falsedefault ensures no breakage for users on older Filebeat versions that don't have the new parser. Users opt in, validate, then the default flips.Goal
Once Phase 1 is stable, evaluate normalizing the
auditdintegration's field output to align with theauditd_managernamespace (auditd.data.*), so that detection rules can target both integrations with a single field reference.Work needed
auditd.data.*namespace (or agree on a shared ECS-aligned namespace)auditd.log.*users (custom rules, dashboards)Stakeholders
Related
--hostfsflag documentation from auditbeat as users migrate to the filestream parser)