We have used Kibana SIEM essentially for network metadata retention, So historically, all network related metadata from various network filebeat modules (suricata, sonicwall, iptables, etc ...) ends up into same indexes for now, which is "filebeat-{version}-{date}-0000X" (0000X because of the ilm and rollover ...).
Since network metadata gets quite huge, we have an ILM policy for "filebeat-" indexes to keep a retention of 30 days (in average its around 100Gb per day ... so around 3Tb total ...).
We are willing now to add to the SIEM the microsoft logs (defender / microsoft threats), but since we want that data to be kept longer and since it also represents less volume, we decided to store it into a new custom index, which is "microsoft-" with storage name "microsoft-{date}-0000X" (0000X because it has ilm rollover enabled). This way, we set up a new lifecycle policy, with more time retention.
We modified in the kibana advanced setting the index listing to use for the SIEM, and we added microsoft-* to it.
Over the SIEM Overview page, we now see events in the " External alert trend" graph, as well as the "Events" graph below it ... BUT NOT in " Detection alert trend" ! and this is wierd because we clearly have alerts inside it ("Data exfiltration to an app that is not sanctioned").
What is also wierd, is that "microsoft.m365_defender.alerts.investigationState" gets tagged "UnsupportedAlertType" ...
Another strange point, if we tell filebeat to store data to legacy "filebeat-" indexes ... it is correctly interpreted, and the event appears in the " Detection alert trend" correctly.
Why is the SIEM not interpreting the alert in the event if using a custom index ? Am I missing something ?
Hi @ulysse31, welcome to our community! We are glad you are using Elastic for SIEM!
One thing to keep in mind is that the Elastic SIEM/Security solution Rules were not automatically updated when you modified the Kibana advanced setting securitySolution:defaultIndex to add microsoft-* to it.
In order to see detection alerts in the detection alert trend histogram at the top of the Overview page (and on the alerts page), you will need to update your detection rules to operate on the data in your microsoft-* indices, by editing the rule and adding microsoft-* to the "Index patterns" section of the rule definition, like this:
Note: if you are using Elastic prebuilt detection rules, you will need to make a duplicate of the rule before you can edit this section. You will need to edit each rule that you want to run on your Microsoft (defender / Microsoft threats) logs.
In a future release, we intend to allow you to change the rule Index patterns setting more easily.
So ... what you are saying ... is that I will need to edit all the SIEM pre-built rules to get it working ...
As convenient as for the prebuild dashboards that I should reedit also myself ...
That suddenly reminds me the time I was with a brand new car ... then ... when a tire bursted because of a rock ... discovered that there were no replacement tire ... but only a tire repair kit instead ... had to call a tow truck because the hole was too large for the repair kit ...
In a way, yes, your answer "addresses" my issue ... but it is clearly not really functional / adapted to real life usage, to say the least ^^'
What would be really helpful would be to have a "common" setting for the prebuild rules, to indicate "default indexes" to use for all rules (or per rule categories) ... because editing each rule ...
Thanks anyways, at least I know to which extend elastic can be magic ^^"
So ... what you are saying ... is that I will need to edit all the SIEM pre-built rules to get it working ...
Currently yes. One way to accomplish this is by exporting the rules, performing a bulk edit in an external editor, and then re-importing them.
Go to the Rules page, select all Elastic rules, and use Bulk actions to duplicate the selected rules.
Select all duplicated rules, and use Bulk actions to export them to an NDJSON file.
Edit the NDJSON file in an external editor of your choosing, for example, replacing "filebeat-*" with "filebeat-*", "microsoft-*"
Import the modified rules using the Import rules button at the top of the rules page. Check the box for "Overwrite existing detection rules with conflicting Rule ID". The modified rules will appear in your rules page as "Custom rules".
Activate the Custom rules of your choosing.
De-activate the "Elastic rules".
What would be really helpful would be to have a "common" setting for the prebuild rules, to indicate "default indexes" to use for all rules (or per rule categories) ... because editing each rule ...
Yes, thanks for that suggestion. We agree, and we are working towards a solution that will allow you to make common changes (such as adding a new index pattern) to a selection of prebuilt rules right within the UI without the need for duplicating and performing external edits.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
