Office 365 Filebeat Module - Improve ECS utilization

Elastic Stack Beats
,
1

We are ingesting O365 data into our Elasticsearch for search, detection in Elastic Security and visualiation through Kibana. However, we have noticed a few areas for improvement within the module. What is most interesting with this module is how data is ingested. The most interesting data related to the events seem to be all placed within the o365.audit.Data field. This makes search and extraction of data from the log source difficult. Ideally the parsing should be done directly in the Filebeat module. We believe there is data within the field that can be used to populate other, more relevant, ECS fields.

Note: we are running filebeat version 8.1.3, but have noticed that none of the newer releases solves our issues.

  • o365.audit.UserKey
    • ECS fields: user.domain | user.name | user.id | user.email | related.user
    • Suggestion:
      • In cases where the kibana.alert.rule.name is “O365 Exchange Suspicious Mailbox Right Delegation” the user.name field is parsed incorrectly. The data will look as follows “domain\user”. It should not include the domain first, this should instead be placed in user.domain and the user.name should only include the actual username. Additionally, the user.email has the same issue where the domain is included before the actual email address. The domain should not be included in the user.email field.
      • In cases where the UserKey field is “SecurityComplianceAlerts” it is not an actual user, but the user.id field is also set to “SecurityComplianceAlerts”. The actual username related to the event can in these cases be extracted from the o365.audit.Data field.
    • o365.audit.UserId
      • ECS fields: user.domain | user.name | user.id | user.email | related.user
      • Suggestion: For record types that are not related to Microsoft Exchange, Azure and SecurityComplianceCenterCommand the UserId field is not an actual user, but a variation of SecurityCompliance*. Data related to the actual user can in these cases be extracted from the o365.audit.Data field.
    • o365.audit.Parameters.User
      • ECS fields: related.user
      • Suggestion: In cases where o365.audit.Workload is “Exchange” the o365.audit.Parameters.User field contains data related to the user on which the action is being performed. Based on this it is relevant to extract the information and include it in the related.user field.
    • o365.audit.Data.f3u
      • ECS fields: user.domain | user.name | user.id | user.email | related.user | related.user
      • Suggestion: In cases where o365.audit.Workload is “SecurityComplianceCenter” and the o365.audit.RecordType is not “24”, the o365.audit.Data.f3u field is set to “SecurityCompliancEvent” and the field does not contain actual user data. This should not populate the ECS fields mentioned above.
    • o365.audit.Data.suid
      • ECS fields: user.domain | user.name | user.id | user.email | related.user | client.user
      • Suggestion: In cases where o365.audit.Workload is “SecurityComplianceCenter”, the o365.audit.RecordType is not “24” and o365.audit.Data.suid is set to either “SecurityComplianceInsights” or “SecurityComplianceEvent” the field does not contain an actual user value. The client.user field should only be populated if the o365.audit.ClientIP field is also populated, this would prevent the field being populated when it is not related to an actual user.
    • o365.audit.Data.isda
      • ECS fields: related.user
      • Suggestion: In cases where o365.audit.Workload is “SecurityComplianceCenter” and the o365.audit.RecordType is not “24”, the o365.audit.Data.isda field contains an array of user objects from o365. These should be used to populate the related.user field.
    • o365.audit.Data.tsd
      • ECS fields: related.user
      • Suggestion: In cases where o365.audit.Workload is “SecurityComplianceCenter” and the o365.audit.RecordType is not “24”, the o365.audit.Data.tsd field contains data related to the sender of an email. This should be used to populate the related.user field. However, sometimes this field can also be malformed and only contain “<>” or a partially parsed utf8 string. Exceptions should be made in these cases.
    • o365.audit.Data.trc
      • ECS fields: user.domain | user.name | user.id | user.email | related.user | client.user
      • Suggestion: In cases where o365.audit.Workload is “SecurityComplianceCenter” and the o365.audit.RecordType is not “24”, the o365.audit.Data.trc field is a recipient of an email. This should parsed and used to populate the mentioned ECS fields. The client.user field should also only be populated if the o365.audit.ClientIP also is populated.
    • o365.audit.Data.zu
      • ECS fields: url.domain | url.extension | url.path | url.original | url.scheme | url.subdomain
      • Suggestion: In all cases where the o365.audit.Data.zu is populated it contains a URL. This data should be parsed and appropriately used to populate the ECS fields mentioned above.
    • o365.audit.Data.reid
      • ECS fields: url.domain | url.extension | url.path | url.original | url.scheme | url.subdomain
      • Suggestion: In cases where o365.audit.Workload is “SecurityComplianceCenter”, the o365.audit.RecordType is not “24” and the o365.audit.Data.zu field is not populated, the o365.audit.Data.reid field may contain URL related data concatenated together with o365.audit.Data.rid data. In cases where the event is related to a file it will not contain a URL(s), but if it is related to a URL it will. In cases where the event is related to a URL the URL should be parsed and appropriately used to populate the ECS fields mentioned above.
    • o365.audit.Data.alk
      • ECS fields: event.url
      • Suggestion: In cases where the o365.audit.Data.alk field is populated, it will contain the URL leading to the actual event. This should populate the event.url field.
    • o365.audit.Parameters.DomainName
      • ECS fields: url.domain | url.extension | url.path | url.original | url.scheme | url.subdomain
      • Suggestion: In cases where the o365.audit.Parameters.DomainName is populated, it will contain a domain. This data should be used to populate the ECS fields mentioned above.
    • o365.audit.Name
      • ECS fields: message | rule.name
      • Suggestion: The message field is currently populated with the contents “New Alert” in events where the o365.audit.Name field exists, while rule.name is populated with the o365.audit.Name data. We would also like to see that the message field populated with the corresponding data.

Additionally, we believe the ECS specification should be improved with the introduction of a new field within the Related fields section. Certain third-party data sources, the O365 module included, send events where multiple URLs are present. An optimal solution would be to add this data to a related.domain or related.url field, none of which currently exist.

2

Per your other thread, you might be better off putting this feature request into an issue on GitHub.

3

I have gone ahead and created a issue (duplicate post) on GitHub as well:

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.