Microsoft Filebeat Module - Lack of ECS utilization

[aforfot (Alexander Forfot)]

We are ingesting Microsoft ATP and M365 Defender data into our Elasticsearch for search, detection in Elastic Security, and visualization through Kibana. However, we have noticed a few specific fields where the Microsoft module does not optimally utilize ECS.

Note: we are running filebeat version 8.1.3, but have noticed that none of the newer releases solves all our issues. Issue #29859 has solved issues related to one specific field, but we still need more improvements.

Microsoft ATP

• microsoft.defender_atp.evidence.userPrincipalName
  • ECS fields: user.email | user.domain | user.name | related.user
  • Suggestion: The userPrincipalName contains the email address of the user. With this it should be possible to populate the user.email field, and also do parsing to extract data into other relevant ECS fields.
• microsoft.defender_atp.loggedOnUsers
  • ECS fields: related.user
  • Suggestion: Populate the related.user field with usernames that are added within the source field based on microsoft.defender_atp.loggedOnUsers.X.accountName
• microsoft.defender_atp.evidence.ipAddress
  • ECS fields: host.ip | related.ip
  • Suggestion: related.ip is populated, but not host.ip. We would like to see this implemented.
• microsoft.defender_atp.evidence.parentProcessFileName
  • ECS fields: file.name | process.name
  • Suggestion: file.name is populated, but not process.name. We would like to see this implemented.
• microsoft.defender_atp.evidence.parentProcessFilePath
  • ECS fields: file.path | process.executable
  • Suggestion: file.path is populated, but not process.executable. We would like to see this implemented.
• file.hash.*, related.hash
  • ECS fields: file.hash.*, process.hash.*, related hash
  • Suggestion: The file.hash.* fields and related hash fields are populated, but not the corresponding process.hash.* fields. We would like to see this implemented.
• message
  • ECS fields: message | rule.name
  • Suggestion: The message field contains the expected data. We would like to see that this data is also populated into the rule.name field.

M365 Defender

• microsoft.m365_defender.incidentUri
  • ECS fields: cloud.account.id
  • Suggestion: Tenant ID is missing in m365_defender documents, but it can be extracted from the incidentUri field based on the value in the tid paramater.
• microsoft.m365_defender.incidentName
  • ECS fields: message | rule.name
  • Suggestion: (1) if the message field is used but incidentName exists, the data in incidentName should replace the data in the message field. (2) if the message is used but incidentName doesn’t exist, the data in the message field can remain the same.
• microsoft.m365_defender.entities.accountName | microsoft.m365_defender.alerts.entities.accountName
  • ECS fields: user.name | user.email | user.domain | related.user
  • Suggestion: user.name and related.user fields contains an email address and not the value specified in the accountName field. The email address should be placed in the user.email field instead and the user.name field populated with the value in from accountName.
• user.id
  • ECS fields: user.id | related.user
  • Suggestion: user.id is populated, but not related user. We would like to see this implemented.
• microsoft.m365_defender.devices.*.loggedOnUsers | microsoft.m365_defender.alerts.devices.*.loggedOnUsers
  • ECS fields: related.user
  • Suggestion: We would like the related.user field to be populated based upon the accountName.
• microsoft.m365_defender.alerts.entities.ipAddress |
  microsoft.m365_defender.entities.ipAddress
  • ECS fields: host.ip | related.ip
  • Suggestion: The IP address is populated in related.ip, but not host.ip. We would like to see this implemented.
• rule.description
  • ECS fields: related.ip
  • Suggestion: The related.ip field can in a lot of cases be populated by other IP adressess that are found within the rule.description field.
• microsoft.m365_defender.entities.parentProcessFilePath
  • ECS fields: file.path | process.executable
  • Suggestion: file.path is populated but not the process.executable field. We would like to see this implemented.
• microsoft.m365_defender.entities.parentProcessFileName
  • ECS fields: file.name | process.name
  • Suggestion: file.name is populated but not the process.name field. We would like to see this implemented.
• file.hash.*, related.hash
  • ECS fields: file.hash.*, process.hash.*, related hash
  • Suggestion: The file.hash.* fields and related hash fields are populated, but not the corresponding process.hash.* fields. We would like to see this implemented.
• microsoft.m365_defender.alerts.entities.detectionStatus
  • ECS fields: event.action
  • Suggestion: If the detectionStatus field is included in the document the event.action field should be populated with the corresponding value.
• microsoft.m365_defender.incidentUri
  • ECS fields: event.url
  • Suggestion: The event.url field should be populated by the data in the .incidentUri field.

[warkolm (Mark Walkom)]

Welcome to our community!

You might be better off putting this feature request into an issue on GitHub.

[aforfot (Alexander Forfot)]

Thanks!

Will look into adding this issue, and the other threads, into GitHub as enhancement requests per your suggestion

[jamie.hynds]

Hi @aforfot - welcome to the community and thank you for taking the time for writing up the detailed feedback. I'm the PM responsible for our security integrations at Elastic. Your timing couldn't be better, as we're actually working on some major enhancements to our M365D integrations, to support Microsoft's new Security Graph API, along with all Defender 365 services. We'll certainly take your ECS mapping feedback onboard and incorporate it into our mappings with this integration update.

The enhancements to the M365D integration will likely be limited to the Elastic Agent integration. You mentioned you are currently running Filebeat. Is there anything blocking you from using the agent integration, once it's available?

[leandrojmp (Leandro Pereira)]

@jamie.hynds

Is there anything blocking you from using the agent integration, once it's available?

I do not have this issue, but just commenting to give some feedback about the Elastic Agent.

My company is planning to move away from Logstash and use Elastic Agent with the integrations, but we felt that Elastic Agent is more limited than Filebeat that already is more limited than Logstash.

One major block for us to use the Elastic Agent is that it does not support Kafka as an input as both Logstash and Filebeat supports, and all our data is consumed by Logstash and Filebeat from Kafka topics.

Of course, is very easy to setup a third-party tool to read from Kafka and send to the Elastic Agente using TCP/UDP, but being able to read from Kafka would increase the Elastic Agent adoption.

[aforfot (Alexander Forfot)]

Hi, and many thanks.

We have also ran into some of the issues as mentioned by @leandrojmp, while additionally not having had the time to review Elastic Agent and its' potential as of yet. It is promising to hear that you are working on major enhancements to the integrations and we will need to look further into Elastic Agent in the near future.

Do you have a rough timeline of when support for the new Security Graph API may be in place for the Elastic Agent? Meanwhile, we still hope that the suggested ECS improvements will be seen for the Filebeat modules in the future as well.

[aforfot (Alexander Forfot)]

I have also opened an issue (duplicate post) on GitHub:

[Filebeat] [Microsoft module] - Lack of ECS utilization