Hi,
I have filebeat 7.10 running with module o365 beat. Similar to how extendedproperties filed for azure active directory workload is parsed. How can i parse this o365.audit.data filed.
the value in this field looks like this:
{"Version":"3.0","VendorName":"Microsoft","ProviderName":"OATP","AlertType":"ea8169fa-0678-4751-8854-aebea7adeceb","StartTimeUtc":"2021-01-19T19:37:54Z","EndTimeUtc":"2021-01-19T19:37:54Z","TimeGenerated":"2021-01-19T20:09:05.023Z","ProcessingEndTime":"2021-01-19T20:17:59.467675Z","Status":"InProgress","Severity":"Informational","ConfidenceLevel":"Unknown","ConfidenceScore":1.0,"IsIncident":false,"ProviderAlertId":"4827709d-0d0e-f576-d600-08d8bcb196e8","SystemAlertId":null,"CorrelationKey":"9762deaa-51ba-4218-af64-ae1932dd8c42","InvestigationIds":["urn:ZappedPhishInvestigation:84ffdeeb9aecd7432b151670ff5ce101"],"Intent":"Probing","ResourceIdentifiers":[{"$id":"1","AadTenantId":"caece813-0c48-4923-8c62-8676bc93605f","Type":"AAD"}],"AzureResourceId":null,"WorkspaceId":null,"WorkspaceSubscriptionId":null,"WorkspaceResourceGroup":null,"AgentId":null,"AlertDisplayName":"Email messages containing phish URLs removed after delivery","Description":"Emails with phish URLs that were delivered and later removed -V1.0.0.5","ExtendedLinks":[{"Href":"https://protection.office.com/viewalerts?id=4827709d-0d0e-f576-d600-08d8bcb196e8","Category":null,"Label":"alert","Type":"webLink"}],"Metadata":{"CustomApps":null,"GenericInfo":null},"Entities":[{"$id":"2","Url":"http://james.boiledpro.com/#amFtZXNAY29hY2hlbGxhLmNvbQ==","Type":"url","ClickCount":0,"EmailCount":1,"Urn":"urn:UrlEntity:2f1b33cf86b82406658d447bbd5bb64a","Source":"OATP","FirstSeen":"2021-01-19T20:12:45"},{"$id":"3","Files":[{"$id":"4","Name":"logo.png","FileHashes":[{"$id":"5","Algorithm":"SHA256","Value":"3B41C7D55985B85B4761BDEEF79CE73207722C0199D7036A337290A01EC26EAC","Type":"filehash"}],"Type":"file","MalwareFamily":""}],"Recipient":"james@contosco.com","Urls":["http://james.boiledpro.com/#amFtZXNAY29hY2hlbGxhLmNvbQ=="],"Threats":["ZapPhish"],"Sender":"support@tap4stuff.com","P1Sender":"010101771bf4fae3-a2418c54-a719-4f81-ad13-f04cf68dd85c-000000@us-west-2.amazonses.com","P1SenderDomain":"us-west-2.amazonses.com","SenderIP":"54.240.27.192","P2Sender":"support@tap4stuff.com","P2SenderDisplayName":"contosco Helpdesk","P2SenderDomain":"tap4stuff.com","ReceivedDate":"2021-01-19T18:42:58","NetworkMessageId":"ccabaffb-fad9-4f2d-b8a4-08d8bcaa09fe","InternetMessageId":"<010101771bf4fae3-a2418c54-a719-4f81-ad13-f04cf68dd85c-000000@us-west-2.amazonses.com>","Subject":"ACTION REQUIRED: New Yearly Update For james@contosco.com","AntispamDirection":"Inbound","DeliveryAction":"Delivered","Language":"en","DeliveryLocation":"Inbox","Type":"mailMessage","Urn":"urn:MailEntity:4bcea468d90fadc679fefb0cf6e6ea93","Source":"OATP","FirstSeen":"2021-01-19T20:12:45"},{"$id":"6","MailboxPrimaryAddress":"james@contosco.com","Upn":"james@contosco.com","RiskLevel":"None","Type":"mailbox","Urn":"urn:UserEntity:775ee0a770248e2050fb52725fc3d615","Source":"OATP","FirstSeen":"2021-01-19T20:12:45"}],"LogCreationTime":"2021-01-19T20:17:59.467675Z","MachineName":"SN1NAM02BG401","SourceTemplateType":"Threat_Single","Category":"ThreatManagement"}
Or like this
o365.audit.data: {"ts":"2021-01-19 04:14:52Z","te":"2021-01-19 04:14:52Z","an":"Custom: Risky Activity from Foreign Countries","ad":"Activity policy 'Custom: Risky Activity from Foreign Countries' was triggered by 'First Last(first.last@contosco.com)'","f3u":"first.last@contosco.com","alk":"https://contosco.portal.cloudappsecurity.com/#/alerts/60065cc0237fdbb8908af067","plk":"https://contosco.portal.cloudappsecurity.com/#/policy/?id=eq(5b076e0df82b1b6d8dfa9f9a,)","mat":"MCAS_ALERT_CABINET_EVENT_MATCH_AUDIT"}
Looking at the original module author
in the section under FAQ that reads, how elastic has integrated this module into filebeat is very different in how the original author performed this activity, i don't know how to apply this script to filebeat to parse this filed correctly.
I would appreciate any guidance on how to parse o365.audt.data filed.
Notes from the module author from its original version:
Can I parse event fields like ExtendedProperties and Parameters that contain arrays of name-value pairs on the client side before shipping them?
As of version 1.5.1, the beat imports the script processor and provides a sample processor script in o365beat.reference.yml to convert fields that contain arrays of name-value pairs into a "normal" object. See this issue for more discussion.
now the script he is referring to is from https://github.com/counteractive/o365beat/blob/master/o365beat.reference.yml
# - script:
# when:
# or:
# - has_fields: ['Parameters']
# - has_fields: ['ExtendedProperties']
# lang: javascript
# id: name_value_array_parser
# source: >
# function process(event){
# var processed = event.Get('processed') || {};
# var parameters = event.Get('Parameters')
# if(!!parameters && !!parameters.length){
# processed.Parameters = processed.Parameters || {};
# for(var i = 0; i < parameters.length; i++){
# var p = parameters[i];
# if(p.Name) processed.Parameters[p.Name] = p.Value;
# }
# }
# var extendedProperties = event.Get('ExtendedProperties')
# if(!!extendedProperties && !!extendedProperties.length){
# processed.ExtendedProperties = processed.ExtendedProperties || {};
# for(var i = 0; i < extendedProperties.length; i++){
# var p = extendedProperties[i];
# if(p.Name) processed.ExtendedProperties[p.Name] = p.Value;
# }
# }
# event.Put('processed', processed);
# }
would really appreciate any help on this.