Filebeat o365 module - Parsing o365.audit.data field

Hi,
I have filebeat 7.10 running with module o365 beat. Similar to how extendedproperties filed for azure active directory workload is parsed. How can i parse this o365.audit.data filed.

the value in this field looks like this:

{"Version":"3.0","VendorName":"Microsoft","ProviderName":"OATP","AlertType":"ea8169fa-0678-4751-8854-aebea7adeceb","StartTimeUtc":"2021-01-19T19:37:54Z","EndTimeUtc":"2021-01-19T19:37:54Z","TimeGenerated":"2021-01-19T20:09:05.023Z","ProcessingEndTime":"2021-01-19T20:17:59.467675Z","Status":"InProgress","Severity":"Informational","ConfidenceLevel":"Unknown","ConfidenceScore":1.0,"IsIncident":false,"ProviderAlertId":"4827709d-0d0e-f576-d600-08d8bcb196e8","SystemAlertId":null,"CorrelationKey":"9762deaa-51ba-4218-af64-ae1932dd8c42","InvestigationIds":["urn:ZappedPhishInvestigation:84ffdeeb9aecd7432b151670ff5ce101"],"Intent":"Probing","ResourceIdentifiers":[{"$id":"1","AadTenantId":"caece813-0c48-4923-8c62-8676bc93605f","Type":"AAD"}],"AzureResourceId":null,"WorkspaceId":null,"WorkspaceSubscriptionId":null,"WorkspaceResourceGroup":null,"AgentId":null,"AlertDisplayName":"Email messages containing phish URLs removed after delivery","Description":"Emails with phish URLs that were delivered and later removed -V1.0.0.5","ExtendedLinks":[{"Href":"https://protection.office.com/viewalerts?id=4827709d-0d0e-f576-d600-08d8bcb196e8","Category":null,"Label":"alert","Type":"webLink"}],"Metadata":{"CustomApps":null,"GenericInfo":null},"Entities":[{"$id":"2","Url":"http://james.boiledpro.com/#amFtZXNAY29hY2hlbGxhLmNvbQ==","Type":"url","ClickCount":0,"EmailCount":1,"Urn":"urn:UrlEntity:2f1b33cf86b82406658d447bbd5bb64a","Source":"OATP","FirstSeen":"2021-01-19T20:12:45"},{"$id":"3","Files":[{"$id":"4","Name":"logo.png","FileHashes":[{"$id":"5","Algorithm":"SHA256","Value":"3B41C7D55985B85B4761BDEEF79CE73207722C0199D7036A337290A01EC26EAC","Type":"filehash"}],"Type":"file","MalwareFamily":""}],"Recipient":"james@contosco.com","Urls":["http://james.boiledpro.com/#amFtZXNAY29hY2hlbGxhLmNvbQ=="],"Threats":["ZapPhish"],"Sender":"support@tap4stuff.com","P1Sender":"010101771bf4fae3-a2418c54-a719-4f81-ad13-f04cf68dd85c-000000@us-west-2.amazonses.com","P1SenderDomain":"us-west-2.amazonses.com","SenderIP":"54.240.27.192","P2Sender":"support@tap4stuff.com","P2SenderDisplayName":"contosco Helpdesk","P2SenderDomain":"tap4stuff.com","ReceivedDate":"2021-01-19T18:42:58","NetworkMessageId":"ccabaffb-fad9-4f2d-b8a4-08d8bcaa09fe","InternetMessageId":"<010101771bf4fae3-a2418c54-a719-4f81-ad13-f04cf68dd85c-000000@us-west-2.amazonses.com>","Subject":"ACTION REQUIRED: New Yearly Update For james@contosco.com","AntispamDirection":"Inbound","DeliveryAction":"Delivered","Language":"en","DeliveryLocation":"Inbox","Type":"mailMessage","Urn":"urn:MailEntity:4bcea468d90fadc679fefb0cf6e6ea93","Source":"OATP","FirstSeen":"2021-01-19T20:12:45"},{"$id":"6","MailboxPrimaryAddress":"james@contosco.com","Upn":"james@contosco.com","RiskLevel":"None","Type":"mailbox","Urn":"urn:UserEntity:775ee0a770248e2050fb52725fc3d615","Source":"OATP","FirstSeen":"2021-01-19T20:12:45"}],"LogCreationTime":"2021-01-19T20:17:59.467675Z","MachineName":"SN1NAM02BG401","SourceTemplateType":"Threat_Single","Category":"ThreatManagement"}

Or like this

o365.audit.data:  	{"ts":"2021-01-19 04:14:52Z","te":"2021-01-19 04:14:52Z","an":"Custom: Risky Activity from Foreign Countries","ad":"Activity policy 'Custom: Risky Activity from Foreign Countries' was triggered by 'First Last(first.last@contosco.com)'","f3u":"first.last@contosco.com","alk":"https://contosco.portal.cloudappsecurity.com/#/alerts/60065cc0237fdbb8908af067","plk":"https://contosco.portal.cloudappsecurity.com/#/policy/?id=eq(5b076e0df82b1b6d8dfa9f9a,)","mat":"MCAS_ALERT_CABINET_EVENT_MATCH_AUDIT"}

Looking at the original module author


in the section under FAQ that reads, how elastic has integrated this module into filebeat is very different in how the original author performed this activity, i don't know how to apply this script to filebeat to parse this filed correctly.

I would appreciate any guidance on how to parse o365.audt.data filed.

Notes from the module author from its original version:

Can I parse event fields like ExtendedProperties and Parameters that contain arrays of name-value pairs on the client side before shipping them?

As of version 1.5.1, the beat imports the script processor and provides a sample processor script in o365beat.reference.yml to convert fields that contain arrays of name-value pairs into a "normal" object. See this issue for more discussion.

now the script he is referring to is from https://github.com/counteractive/o365beat/blob/master/o365beat.reference.yml

 # - script:
  #     when:
  #       or:
  #         - has_fields: ['Parameters']
  #         - has_fields: ['ExtendedProperties']
  #     lang: javascript
  #     id: name_value_array_parser
  #     source: >
  #       function process(event){
  #         var processed = event.Get('processed') || {};
  #         var parameters = event.Get('Parameters')
  #         if(!!parameters && !!parameters.length){
  #           processed.Parameters = processed.Parameters || {};
  #           for(var i = 0; i < parameters.length; i++){
  #             var p = parameters[i];
  #             if(p.Name) processed.Parameters[p.Name] = p.Value;
  #           }
  #         }
  #         var extendedProperties = event.Get('ExtendedProperties')
  #         if(!!extendedProperties && !!extendedProperties.length){
  #           processed.ExtendedProperties = processed.ExtendedProperties || {};
  #           for(var i = 0; i < extendedProperties.length; i++){
  #             var p = extendedProperties[i];
  #             if(p.Name) processed.ExtendedProperties[p.Name] = p.Value;
  #           }
  #         }
  #         event.Put('processed', processed);
  #       }

would really appreciate any help on this.

I think that using a script processor for this fields is not a bad idea. You can give a try to JSON processor (ingest node): https://www.elastic.co/guide/en/elasticsearch/reference/current/json-processor.html

Thanks Marcin, i am looking to leverage the script processor as documented in the original o365 beat, the JSON processor wouldn't work for some of the other logs that get stored in the o365.audit.data field. I have no idea how to go about it with the filebeat.

Thank you

Anyone have any ideas how i can parse the o365.audit.data fiiled in filebeat?

@jamie.hynds
Hi Jamie , do you have any insights on this by any chance or know someone that might thanks .