Skip to content

[santa] update ingest pipeline to match new log format #3347

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
efd6 merged 6 commits into from
May 18, 2022
Merged

[santa] update ingest pipeline to match new log format #3347

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
fab226c
update log samples for Santa based on version 2022.4
cl-damiapoquet May 12, 2022
dda224f
modify ingest pipeline and fields to match with Santa logs
cl-damiapoquet May 12, 2022
d7850eb
update README.md
cl-damiapoquet May 12, 2022
4ca682b
update documentation
cl-damiapoquet May 16, 2022
575dd1d
mark release as breaking-change
cl-damiapoquet May 16, 2022
f4f44c6
change pidversion field and package version description
cl-damiapoquet May 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
change pidversion field and package version description
  • Loading branch information
@cl-damiapoquet
cl-damiapoquet committed May 17, 2022
commit f4f44c604be8066e7e54277a122b7b8b23bce067
4 changes: 2 additions & 2 deletions packages/santa/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# newer versions go on top
- version: "2.2.0"
- version: "3.0.0"
changes:
- description: Update log format
- description: Update log format to support the GA releases of Santa. The pre-GA Santa log format (circa 2017) is no longer accepted.
type: breaking-change
link: https://github.com/elastic/integrations/pull/3347
- version: "2.1.0"
Expand Down
20 changes: 10 additions & 10 deletions packages/santa/data_stream/log/_dev/test/pipeline/test-santa-raw.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,6 @@
"pid": 1
},
"pid": 71993,
"pidversion": 1097732,
"start": "2022-05-12T11:38:03.923Z"
},
"related": {
Expand All @@ -66,6 +65,7 @@
"decision": "ALLOW",
"explain": "critical system binary",
"mode": "M",
"pidversion": 1097732,
"reason": "BINARY"
},
"tags": [
Expand Down Expand Up @@ -120,7 +120,6 @@
"pid": 1
},
"pid": 72012,
"pidversion": 1097765,
"start": "2022-05-12T11:38:42.781Z"
},
"related": {
Expand All @@ -140,6 +139,7 @@
},
"decision": "ALLOW",
"mode": "M",
"pidversion": 1097765,
"reason": "CERT"
},
"tags": [
Expand Down Expand Up @@ -180,7 +180,6 @@
"pid": 1
},
"pid": 377,
"pidversion": 833,
"start": "2022-05-12T11:33:56.696Z"
},
"related": {
Expand All @@ -189,7 +188,8 @@
]
},
"santa": {
"action": "DELETE"
"action": "DELETE",
"pidversion": 833
},
"tags": [
"preserve_original_event"
Expand Down Expand Up @@ -230,7 +230,6 @@
"pid": 1
},
"pid": 71559,
"pidversion": 1096716,
"start": "2022-05-12T11:30:05.248Z"
},
"related": {
Expand All @@ -239,7 +238,8 @@
]
},
"santa": {
"action": "LINK"
"action": "LINK",
"pidversion": 1096716
},
"tags": [
"preserve_original_event"
Expand Down Expand Up @@ -280,7 +280,6 @@
"pid": 1
},
"pid": 546,
"pidversion": 1285,
"start": "2022-05-12T11:30:16.125Z"
},
"related": {
Expand All @@ -289,7 +288,8 @@
]
},
"santa": {
"action": "RENAME"
"action": "RENAME",
"pidversion": 1285
},
"tags": [
"preserve_original_event"
Expand Down Expand Up @@ -329,7 +329,6 @@
"pid": 0
},
"pid": 1,
"pidversion": 521,
"start": "2022-05-12T11:38:05.278Z"
},
"related": {
Expand All @@ -338,7 +337,8 @@
]
},
"santa": {
"action": "WRITE"
"action": "WRITE",
"pidversion": 521
},
"tags": [
"preserve_original_event"
Expand Down
4 changes: 2 additions & 2 deletions packages/santa/data_stream/log/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ processors:
- grok:
field: event.original
patterns:
- '\[%{TIMESTAMP_ISO8601:process.start}\] %{NOT_SEPARATOR:log.level} santad: action=%{NOT_SEPARATOR:santa.action}\|decision=%{NOT_SEPARATOR:santa.decision}\|reason=%{NOT_SEPARATOR:santa.reason}(\|explain=%{NOT_SEPARATOR:santa.explain})?\|sha256=%{NOT_SEPARATOR:process.hash.sha256}(\|cert_sha256=%{NOT_SEPARATOR:santa.certificate.sha256})?(\|cert_cn=%{NOT_SEPARATOR:santa.certificate.common_name})?\|pid=%{NUMBER:process.pid:long}\|pidversion=%{NUMBER:process.pidversion:long}\|ppid=%{NUMBER:process.parent.pid:long}\|uid=%{NUMBER:user.id}\|user=%{NOT_SEPARATOR:user.name}\|gid=%{NUMBER:group.id}\|group=%{NOT_SEPARATOR:group.name}\|mode=%{WORD:santa.mode}\|path=%{NOT_SEPARATOR:process.executable}(\|args=%{NOT_SEPARATOR:santa.args})?'
- '\[%{TIMESTAMP_ISO8601:process.start}\] %{NOT_SEPARATOR:log.level} santad: action=%{NOT_SEPARATOR:santa.action}\|path=%{NOT_SEPARATOR:file.path}(\|newpath=%{NOT_SEPARATOR:file.target_path})?\|pid=%{NUMBER:process.pid:long}\|pidversion=%{NUMBER:process.pidversion:long}\|ppid=%{NUMBER:process.parent.pid:long}\|process=%{NOT_SEPARATOR:process.name}\|processpath=%{NOT_SEPARATOR:process.executable}\|uid=%{NUMBER:user.id}\|user=%{NOT_SEPARATOR:user.name}\|gid=%{NUMBER:group.id}\|group=%{NOT_SEPARATOR:group.name}'
- '\[%{TIMESTAMP_ISO8601:process.start}\] %{NOT_SEPARATOR:log.level} santad: action=%{NOT_SEPARATOR:santa.action}\|decision=%{NOT_SEPARATOR:santa.decision}\|reason=%{NOT_SEPARATOR:santa.reason}(\|explain=%{NOT_SEPARATOR:santa.explain})?\|sha256=%{NOT_SEPARATOR:process.hash.sha256}(\|cert_sha256=%{NOT_SEPARATOR:santa.certificate.sha256})?(\|cert_cn=%{NOT_SEPARATOR:santa.certificate.common_name})?\|pid=%{NUMBER:process.pid:long}\|pidversion=%{NUMBER:santa.pidversion:long}\|ppid=%{NUMBER:process.parent.pid:long}\|uid=%{NUMBER:user.id}\|user=%{NOT_SEPARATOR:user.name}\|gid=%{NUMBER:group.id}\|group=%{NOT_SEPARATOR:group.name}\|mode=%{WORD:santa.mode}\|path=%{NOT_SEPARATOR:process.executable}(\|args=%{NOT_SEPARATOR:santa.args})?'
- '\[%{TIMESTAMP_ISO8601:process.start}\] %{NOT_SEPARATOR:log.level} santad: action=%{NOT_SEPARATOR:santa.action}\|path=%{NOT_SEPARATOR:file.path}(\|newpath=%{NOT_SEPARATOR:file.target_path})?\|pid=%{NUMBER:process.pid:long}\|pidversion=%{NUMBER:santa.pidversion:long}\|ppid=%{NUMBER:process.parent.pid:long}\|process=%{NOT_SEPARATOR:process.name}\|processpath=%{NOT_SEPARATOR:process.executable}\|uid=%{NUMBER:user.id}\|user=%{NOT_SEPARATOR:user.name}\|gid=%{NUMBER:group.id}\|group=%{NOT_SEPARATOR:group.name}'
- '\[%{TIMESTAMP_ISO8601:timestamp}\] %{NOT_SEPARATOR:log.level} santad: action=%{NOT_SEPARATOR:santa.action}\|mount=%{NOT_SEPARATOR:santa.disk.mount}?\|volume=%{NOT_SEPARATOR:santa.disk.volume}\|bsdname=%{NOT_SEPARATOR:santa.disk.bsdname}?(\|fs=%{NOT_SEPARATOR:santa.disk.fs})?(\|model=%{NOT_SEPARATOR:santa.disk.model}?)?(\|serial=%{NOT_SEPARATOR:santa.disk.serial}?)?(\|bus=%{NOT_SEPARATOR:santa.disk.bus}?)?(\|dmgpath=%{NOT_SEPARATOR:santa.disk.dmgpath}?)?(\|appearance=%{TIMESTAMP_ISO8601:santa.disk.appearance})?'
pattern_definitions:
NOT_SEPARATOR: '[^\|]+'
Expand Down
3 changes: 3 additions & 0 deletions packages/santa/data_stream/log/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,6 @@
- name: certificate.sha256
type: keyword
description: SHA256 hash of code signing certificate.
- name: pidversion
type: long
description: macOS process identity version.
2 changes: 1 addition & 1 deletion packages/santa/data_stream/log/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@
"pid": 1
},
"pid": 71993,
"pidversion": 1097732,
"start": "2022-05-12T11:38:03.923Z"
},
"related": {
Expand All @@ -64,6 +63,7 @@
"decision": "ALLOW",
"explain": "critical system binary",
"mode": "M",
"pidversion": 1097732,
"reason": "BINARY"
},
"tags": [
Expand Down
3 changes: 2 additions & 1 deletion packages/santa/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,6 @@ An example event for `log` looks as following:
"pid": 1
},
"pid": 71993,
"pidversion": 1097732,
"start": "2022-05-12T11:38:03.923Z"
},
"related": {
Expand All @@ -86,6 +85,7 @@ An example event for `log` looks as following:
"decision": "ALLOW",
"explain": "critical system binary",
"mode": "M",
"pidversion": 1097732,
"reason": "BINARY"
},
"tags": [
Expand Down Expand Up @@ -176,6 +176,7 @@ An example event for `log` looks as following:
| santa.disk.volume | The volume name. | keyword |
| santa.explain | Further details for the decision. | keyword |
| santa.mode | Operating mode of Santa. | keyword |
| santa.pidversion | macOS process identity version. | long |
| santa.reason | Reason for the decision. | keyword |
| tags | List of keywords used to tag each event. | keyword |
| user.id | Unique identifier of the user. | keyword |
Expand Down
2 changes: 1 addition & 1 deletion packages/santa/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name: santa
title: Google Santa Logs
version: 2.2.0
version: 3.0.0
release: ga
description: Collect and parse logs from Google Santa instances with Elastic Agent.
type: integration
Expand Down