Skip to content

[abnormal_security] Initial release of the Abnormal Security #10653

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
efd6 merged 5 commits into from
Aug 9, 2024
Merged

[abnormal_security] Initial release of the Abnormal Security #10653

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
bb0c4e0
Initial release
brijesh-elastic Jul 30, 2024
33f8bfc
Update changelog entry
brijesh-elastic Jul 30, 2024
ea29f81
Resolve comments
brijesh-elastic Aug 6, 2024
9e9e159
Resolve comments
brijesh-elastic Aug 8, 2024
bbd36c3
Fix indent
brijesh-elastic Aug 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Resolve comments
  • Loading branch information
@brijesh-elastic
brijesh-elastic committed Aug 8, 2024
commit 9e9e159d5c948c765a50da0809f0fb03a9b1a5e1
22 changes: 13 additions & 9 deletions ..._stream/ai_security_mailbox/_dev/test/pipeline/test-ai-security-mailbox.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@
"preserve_duplicate_custom_fields"
],
"threat": {
"technique": {
"tactic": {
"name": [
"Attack Type: Graymail"
]
Expand Down Expand Up @@ -124,7 +124,7 @@
"threat"
],
"id": "8369181238656832368",
"kind": "event",
"kind": "enrichment",
"original": "{\"campaignId\":\"b4849d52-471b-396f-b737-775826c7e557\",\"firstReported\":\"2024-07-29T06:01:27Z\",\"lastReported\":\"2024-07-29T06:01:27Z\",\"messageId\":\"8369181238656832368\",\"subject\":\"Important document\",\"fromName\":\"john\",\"fromAddress\":\"john@example.com\",\"recipientName\":\"bob\",\"recipientAddress\":\"bob@example.com\",\"judgementStatus\":\"Spam\",\"overallStatus\":\"\",\"attackType\":\"Attack Type: Spam\"}",
"type": [
"indicator"
Expand All @@ -150,9 +150,10 @@
"email": {
"address": "john@example.com"
},
"name": "john@example.com",
"type": "email-addr"
},
"technique": {
"tactic": {
"name": [
"Attack Type: Spam"
Comment thread
kcreddy marked this conversation as resolved.
Resolved
]
Expand Down Expand Up @@ -211,7 +212,7 @@
"threat"
],
"id": "-4661319438952668559",
"kind": "event",
"kind": "enrichment",
"original": "{\"campaignId\":\"2e1354de-f9cb-3192-82ff-714204358eb7\",\"firstReported\":\"2024-04-29T13:45:00Z\",\"lastReported\":\"2024-07-27T02:43:12Z\",\"messageId\":\"-4661319438952668559\",\"subject\":\"Your statement is available\",\"fromName\":\"bob\",\"fromAddress\":\"bob@example.com\",\"recipientName\":\"john\",\"recipientAddress\":\"john@example.com\",\"judgementStatus\":\"Spam\",\"overallStatus\":\"\",\"attackType\":\"Attack Type: Spam\"}",
"type": [
"indicator"
Expand All @@ -237,9 +238,10 @@
"email": {
"address": "bob@example.com"
},
"name": "bob@example.com",
"type": "email-addr"
},
"technique": {
"tactic": {
"name": [
"Attack Type: Spam"
]
Expand Down Expand Up @@ -300,7 +302,7 @@
"threat"
],
"id": "-7731344566782869529",
"kind": "event",
"kind": "enrichment",
"original": "{\"campaignId\":\"8978dff3-7be5-3ab4-a841-30b90fc5788d\",\"firstReported\":\"2024-08-01T20:19:24Z\",\"lastReported\":\"2024-08-01T20:19:24Z\",\"messageId\":\"-7731344566782869529\",\"subject\":\"New Secure Message Scanned and sent\",\"fromName\":\"john@example.com\",\"fromAddress\":\"john@example.com\",\"recipientName\":\"bob\",\"recipientAddress\":\"bob@example.com\",\"judgementStatus\":\"Malicious\",\"overallStatus\":\"Auto-Remediated\",\"attackType\":\"Attack Type: Scam\"}",
"type": [
"indicator"
Expand All @@ -325,9 +327,10 @@
"email": {
"address": "john@example.com"
},
"name": "john@example.com",
"type": "email-addr"
},
"technique": {
"tactic": {
"name": [
"Attack Type: Scam"
]
Expand Down Expand Up @@ -388,7 +391,7 @@
"threat"
],
"id": "7372776983204655999",
"kind": "event",
"kind": "enrichment",
"original": "{\"campaignId\":\"baee573f-4a06-3791-9ed7-4f5153a7fe5d\",\"firstReported\":\"2024-07-30T15:51:48Z\",\"lastReported\":\"2024-07-30T15:51:48Z\",\"messageId\":\"7372776983204655999\",\"subject\":\"Inform Tech Team\",\"fromName\":\"john\",\"fromAddress\":\"john@example.com\",\"recipientName\":\"bob\",\"recipientAddress\":\"bob@example.com\",\"judgementStatus\":\"Malicious\",\"overallStatus\":\"Auto-Remediated\",\"attackType\":\"Attack Type: Phishing: Credential\"}",
"type": [
"indicator"
Expand All @@ -414,9 +417,10 @@
"email": {
"address": "john@example.com"
},
"name": "john@example.com",
"type": "email-addr"
},
"technique": {
"tactic": {
"name": [
"Attack Type: Phishing: Credential"
]
Expand Down
16 changes: 13 additions & 3 deletions ...normal_security/data_stream/ai_security_mailbox/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,8 @@ processors:
target_field: abnormal_security.ai_security_mailbox.attack.type
ignore_missing: true
- append:
field: threat.technique.name
tag: append_abnormal_security_ai_security_mailbox_attack_type_into_threat_technique_name
field: threat.tactic.name
tag: append_abnormal_security_ai_security_mailbox_attack_type_into_threat_tactic_name
value: '{{{abnormal_security.ai_security_mailbox.attack.type}}}'
allow_duplicates: false
if: ctx.abnormal_security?.ai_security_mailbox?.attack?.type != null
Expand Down Expand Up @@ -100,6 +100,11 @@ processors:
tag: rename_judgementStatus
target_field: abnormal_security.ai_security_mailbox.judgement_status
ignore_missing: true
- set:
field: event.kind
tag: set_event_kind_to_enrichment
value: enrichment
if: ctx.abnormal_security?.ai_security_mailbox?.judgement_status != null && ['spam','malicious'].contains(ctx.abnormal_security.ai_security_mailbox.judgement_status.toLowerCase())
- append:
field: event.category
tag: append_event_category
Expand Down Expand Up @@ -128,7 +133,12 @@ processors:
if: ctx.abnormal_security?.ai_security_mailbox?.from?.address != null
- set:
field: threat.indicator.email.address
tag: set_event_kind_to_event
tag: set_threat_indicator_email_address_to_from_address
copy_from: abnormal_security.ai_security_mailbox.from.address
if: ctx.abnormal_security?.ai_security_mailbox?.judgement_status != null && ['spam','malicious'].contains(ctx.abnormal_security.ai_security_mailbox.judgement_status.toLowerCase())
- set:
field: threat.indicator.name
tag: set_threat_indicator_name_to_from_address
copy_from: abnormal_security.ai_security_mailbox.from.address
if: ctx.abnormal_security?.ai_security_mailbox?.judgement_status != null && ['spam','malicious'].contains(ctx.abnormal_security.ai_security_mailbox.judgement_status.toLowerCase())
- append:
Expand Down
12 changes: 6 additions & 6 deletions packages/abnormal_security/data_stream/ai_security_mailbox/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,15 @@
}
},
"agent": {
"ephemeral_id": "becef71f-2db4-4fa6-92f2-773b45c2f1bc",
"id": "7aaba523-565c-4597-bc42-59135436336b",
"ephemeral_id": "cafadbdd-dc09-45ac-aec4-49d7250ebd32",
"id": "9783be93-6fa9-44ba-8f6d-eda7dcb99151",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.13.0"
},
"data_stream": {
"dataset": "abnormal_security.ai_security_mailbox",
"namespace": "72265",
"namespace": "38204",
"type": "logs"
},
"destination": {
Expand All @@ -43,7 +43,7 @@
"version": "8.11.0"
},
"elastic_agent": {
"id": "7aaba523-565c-4597-bc42-59135436336b",
"id": "9783be93-6fa9-44ba-8f6d-eda7dcb99151",
"snapshot": false,
"version": "8.13.0"
},
Expand All @@ -64,7 +64,7 @@
"agent_id_status": "verified",
"dataset": "abnormal_security.ai_security_mailbox",
"id": "7063250485337877109",
"ingested": "2024-08-06T09:40:30Z",
"ingested": "2024-08-08T05:41:05Z",
"kind": "event",
"original": "{\"attackType\":\"Attack Type: Graymail\",\"campaignId\":\"fff51768-c446-34e1-97a8-9802c29c3ebd\",\"firstReported\":\"2024-07-26T10:30:06Z\",\"fromAddress\":\"bob@example.com\",\"fromName\":\"bob@example.com\",\"judgementStatus\":\"Safe\",\"lastReported\":\"2024-07-26T10:30:06Z\",\"messageId\":\"7063250485337877109\",\"overallStatus\":\"No Action Needed\",\"recipientAddress\":\"john@example.com\",\"recipientName\":\"john\",\"subject\":\"Days of Understanding 2024\"}"
},
Expand All @@ -89,7 +89,7 @@
"abnormal_security-ai_security_mailbox"
],
"threat": {
"technique": {
"tactic": {
"name": [
"Attack Type: Graymail"
]
Expand Down
5 changes: 5 additions & 0 deletions ...ges/abnormal_security/data_stream/threat/_dev/test/pipeline/test-threat.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@
"email": {
"address": "john@example.com"
},
"name": "john@example.com",
"reference": "https://portal.abnormalsecurity.com/home/threat-center/remediation-history/-4898763529198204348",
"type": "email-addr"
},
Expand Down Expand Up @@ -268,6 +269,7 @@
"email": {
"address": "john@example.com"
},
"name": "john@example.com",
"reference": "https://portal.abnormalsecurity.com/home/threat-center/remediation-history/12345",
"type": "email-addr"
},
Expand Down Expand Up @@ -414,6 +416,7 @@
"email": {
"address": "bob@example.com"
},
"name": "bob@example.com",
"reference": "https://portal.abnormalsecurity.com/home/threat-center/remediation-history/9876543",
"type": "email-addr"
},
Expand Down Expand Up @@ -550,6 +553,7 @@
"email": {
"address": "alias@example.com"
},
"name": "alias@example.com",
"reference": "https://portal.abnormalsecurity.com/home/threat-center/remediation-history/3456787654",
"type": "email-addr"
},
Expand Down Expand Up @@ -689,6 +693,7 @@
"email": {
"address": "john@example.com"
},
"name": "john@example.com",
"reference": "https://portal.abnormalsecurity.com/home/threat-center/remediation-history/3456765434567654",
"type": "email-addr"
},
Expand Down
5 changes: 5 additions & 0 deletions packages/abnormal_security/data_stream/threat/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,11 @@ processors:
tag: set_threat_indicator_email_address_from_threat_from_address
copy_from: abnormal_security.threat.from_address
ignore_empty_value: true
- set:
field: threat.indicator.name
tag: set_threat_indicator_name_from_threat_from_address
copy_from: abnormal_security.threat.from_address
ignore_empty_value: true
- append:
field: related.user
tag: append_from_address_into_related_user
Expand Down
7 changes: 4 additions & 3 deletions packages/abnormal_security/data_stream/threat/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,15 +45,15 @@
}
},
"agent": {
"ephemeral_id": "7a47e772-7565-4c03-be8a-43a442be5c92",
"ephemeral_id": "3cfaa9dc-bca8-4e29-a807-77b68709b731",
"id": "7aaba523-565c-4597-bc42-59135436336b",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.13.0"
},
"data_stream": {
"dataset": "abnormal_security.threat",
"namespace": "63490",
"namespace": "37330",
"type": "logs"
},
"ecs": {
Expand Down Expand Up @@ -88,7 +88,7 @@
],
"dataset": "abnormal_security.threat",
"id": "2260288475997441000",
"ingested": "2024-08-06T09:43:49Z",
"ingested": "2024-08-08T06:53:48Z",
"kind": "enrichment",
"original": "{\"abxMessageId\":2260288475997441000,\"abxPortalUrl\":\"https://portal.abnormalsecurity.com/home/threat-center/remediation-history/3456765434567654\",\"attachmentCount\":0,\"attachmentNames\":[],\"attackStrategy\":\"Unknown Sender\",\"attackType\":\"Spam\",\"attackVector\":\"Link\",\"attackedParty\":\"Employee (Other)\",\"autoRemediated\":true,\"ccEmails\":[],\"fromAddress\":\"john@example.com\",\"fromName\":\"john\",\"impersonatedParty\":\"None / Others\",\"internetMessageId\":\"\\u003cAZz8NUMEST-qmuz77_koic@example\\u003e\",\"isRead\":false,\"postRemediated\":false,\"receivedTime\":\"2024-07-17T23:25:38Z\",\"recipientAddress\":\"bob@example.com\",\"remediationStatus\":\"Auto-Remediated\",\"remediationTimestamp\":\"2024-07-17T23:25:45.73564Z\",\"replyToEmails\":[],\"returnPath\":\"bounce-bob_H181S7GUCF@example.com\",\"senderDomain\":\"example.com\",\"senderIpAddress\":\"81.2.69.142\",\"sentTime\":\"2024-07-17T23:25:29Z\",\"subject\":\"YoU.have.𝗪𝟬0𝗡𝗡 a K0baIt 215-piece_ToooI_Set_Noo0wW..#GBOB\",\"summaryInsights\":[\"Abnormal Email Body HTML\",\"Invisible characters found in Email\",\"Suspicious Link\",\"Unusual Sender\",\"Unusual Sender Domain\"],\"threatId\":\"bf255f2d-a2ad-3f50-5075-fdcc24308bbd\",\"toAddresses\":[\"bob@example.com\"],\"urlCount\":1,\"urls\":[\"https://www.example.com/\"]}",
"reference": "https://portal.abnormalsecurity.com/home/threat-center/remediation-history/3456765434567654",
Expand Down Expand Up @@ -145,6 +145,7 @@
"email": {
"address": "john@example.com"
},
"name": "john@example.com",
"reference": "https://portal.abnormalsecurity.com/home/threat-center/remediation-history/3456765434567654",
"type": "email-addr"
},
Expand Down
19 changes: 10 additions & 9 deletions packages/abnormal_security/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,15 +98,15 @@ An example event for `ai_security_mailbox` looks as following:
}
},
"agent": {
"ephemeral_id": "becef71f-2db4-4fa6-92f2-773b45c2f1bc",
"id": "7aaba523-565c-4597-bc42-59135436336b",
"ephemeral_id": "cafadbdd-dc09-45ac-aec4-49d7250ebd32",
"id": "9783be93-6fa9-44ba-8f6d-eda7dcb99151",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.13.0"
},
"data_stream": {
"dataset": "abnormal_security.ai_security_mailbox",
"namespace": "72265",
"namespace": "38204",
"type": "logs"
},
"destination": {
Expand All @@ -118,7 +118,7 @@ An example event for `ai_security_mailbox` looks as following:
"version": "8.11.0"
},
"elastic_agent": {
"id": "7aaba523-565c-4597-bc42-59135436336b",
"id": "9783be93-6fa9-44ba-8f6d-eda7dcb99151",
"snapshot": false,
"version": "8.13.0"
},
Expand All @@ -139,7 +139,7 @@ An example event for `ai_security_mailbox` looks as following:
"agent_id_status": "verified",
"dataset": "abnormal_security.ai_security_mailbox",
"id": "7063250485337877109",
"ingested": "2024-08-06T09:40:30Z",
"ingested": "2024-08-08T05:41:05Z",
"kind": "event",
"original": "{\"attackType\":\"Attack Type: Graymail\",\"campaignId\":\"fff51768-c446-34e1-97a8-9802c29c3ebd\",\"firstReported\":\"2024-07-26T10:30:06Z\",\"fromAddress\":\"bob@example.com\",\"fromName\":\"bob@example.com\",\"judgementStatus\":\"Safe\",\"lastReported\":\"2024-07-26T10:30:06Z\",\"messageId\":\"7063250485337877109\",\"overallStatus\":\"No Action Needed\",\"recipientAddress\":\"john@example.com\",\"recipientName\":\"john\",\"subject\":\"Days of Understanding 2024\"}"
},
Expand All @@ -164,7 +164,7 @@ An example event for `ai_security_mailbox` looks as following:
"abnormal_security-ai_security_mailbox"
],
"threat": {
"technique": {
"tactic": {
"name": [
"Attack Type: Graymail"
]
Expand Down Expand Up @@ -498,15 +498,15 @@ An example event for `threat` looks as following:
}
},
"agent": {
"ephemeral_id": "7a47e772-7565-4c03-be8a-43a442be5c92",
"ephemeral_id": "3cfaa9dc-bca8-4e29-a807-77b68709b731",
"id": "7aaba523-565c-4597-bc42-59135436336b",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.13.0"
},
"data_stream": {
"dataset": "abnormal_security.threat",
"namespace": "63490",
"namespace": "37330",
"type": "logs"
},
"ecs": {
Expand Down Expand Up @@ -541,7 +541,7 @@ An example event for `threat` looks as following:
],
"dataset": "abnormal_security.threat",
"id": "2260288475997441000",
"ingested": "2024-08-06T09:43:49Z",
"ingested": "2024-08-08T06:53:48Z",
"kind": "enrichment",
"original": "{\"abxMessageId\":2260288475997441000,\"abxPortalUrl\":\"https://portal.abnormalsecurity.com/home/threat-center/remediation-history/3456765434567654\",\"attachmentCount\":0,\"attachmentNames\":[],\"attackStrategy\":\"Unknown Sender\",\"attackType\":\"Spam\",\"attackVector\":\"Link\",\"attackedParty\":\"Employee (Other)\",\"autoRemediated\":true,\"ccEmails\":[],\"fromAddress\":\"john@example.com\",\"fromName\":\"john\",\"impersonatedParty\":\"None / Others\",\"internetMessageId\":\"\\u003cAZz8NUMEST-qmuz77_koic@example\\u003e\",\"isRead\":false,\"postRemediated\":false,\"receivedTime\":\"2024-07-17T23:25:38Z\",\"recipientAddress\":\"bob@example.com\",\"remediationStatus\":\"Auto-Remediated\",\"remediationTimestamp\":\"2024-07-17T23:25:45.73564Z\",\"replyToEmails\":[],\"returnPath\":\"bounce-bob_H181S7GUCF@example.com\",\"senderDomain\":\"example.com\",\"senderIpAddress\":\"81.2.69.142\",\"sentTime\":\"2024-07-17T23:25:29Z\",\"subject\":\"YoU.have.𝗪𝟬0𝗡𝗡 a K0baIt 215-piece_ToooI_Set_Noo0wW..#GBOB\",\"summaryInsights\":[\"Abnormal Email Body HTML\",\"Invisible characters found in Email\",\"Suspicious Link\",\"Unusual Sender\",\"Unusual Sender Domain\"],\"threatId\":\"bf255f2d-a2ad-3f50-5075-fdcc24308bbd\",\"toAddresses\":[\"bob@example.com\"],\"urlCount\":1,\"urls\":[\"https://www.example.com/\"]}",
"reference": "https://portal.abnormalsecurity.com/home/threat-center/remediation-history/3456765434567654",
Expand Down Expand Up @@ -598,6 +598,7 @@ An example event for `threat` looks as following:
"email": {
"address": "john@example.com"
},
"name": "john@example.com",
"reference": "https://portal.abnormalsecurity.com/home/threat-center/remediation-history/3456765434567654",
"type": "email-addr"
},
Expand Down
4 changes: 2 additions & 2 deletions ...mal_security/kibana/dashboard/abnormal_security-a4364503-ada3-4fe6-a054-d152accf207c.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
"feec41a2-407a-4b96-9f3b-0d249cce7b9d": {
"explicitInput": {
"enhancements": {},
"fieldName": "threat.technique.name",
"fieldName": "threat.tactic.name",
"grow": true,
"id": "feec41a2-407a-4b96-9f3b-0d249cce7b9d",
"searchTechnique": "prefix",
Expand Down Expand Up @@ -406,7 +406,7 @@
"size": 10
},
"scale": "ordinal",
"sourceField": "threat.technique.name"
"sourceField": "threat.tactic.name"
}
},
"incompleteColumns": {},
Expand Down