Are there any pans to add alerting to the SIEM app? Specifically the ability to create a watch as part of the Detection Rule setup?
I see others have requested allowing custom Signals columns to be saved, ideally I would also want this but I think the default columns should be changed. Personally I would remove Version, Method, Severity and event.module so the actual signal data is surfaced in the Signals pane. External Alerts columns should be changed to include the rule.reference column, the Endgame data that's surfaced here is not that useful but having the link to the alert in the endgame console would be very useful.
The Network Map in version 7.6.2 appears to have changed to use the endgame-* indices and I don't see a way to change this. This renders the map pretty much useless in our case where the actual external geo data is in the proxy and firewall logs that we ingest and not from endpoints. Can we get an option to customize the data sources or layers for this or at least revert back to normal index geo points?
Can the Timeline pane be moved to a new TAB, having it render in the same page as the SIEM data you want to investigate makes the interface very clunky and limits the appeal of using timelines, if this was opened in a new tab that could be moved to a second screen would make it much better.
Can a summary of the ML job anomalies be surfaced on the SIEM Overview page, even the same page that you get from the ML overview page would be fine.
The app itself has real potential (we've found the detection aspect brilliant and pretty much replaces a lot of our previous manual discover and some ML jobs), with some UI tweaks, the addition of alerting directly from the SIEM app and the ability to customise the layout a bit it could be really good.
Hey there Gareth, thanks for the kind words, and glad you've been getting some use out of the Detection Engine and SIEM App as a whole. We definitely have some UI tweaks lined up, and I would highly encourage you to open feature requests for those not covered, as that helps us prioritize upcoming features.
That said, let's see if we can't get you some answers
Are there any pans to add alerting to the SIEM app? Specifically the ability to create a watch as part of the Detection Rule setup?
I see others have requested allowing custom Signals columns to be saved, ideally I would also want this but I think the default columns should be changed. Personally I would remove Version, Method, Severity and event.module so the actual signal data is surfaced in the Signals pane. External Alerts columns should be changed to include the rule.reference column, the Endgame data that's surfaced here is not that useful but having the link to the alert in the endgame console would be very useful.
Thanks for the feedback here! We're going to be improving the Signals/Alerts view to highlight the actual signal data as you point out, and enabling the persistence of user-defined columns is definitely one of the ways we'll be looking to do this. The Detections Engine is still in beta, so feedback like this is definitely appreciated.
The Network Map in version 7.6.2 appears to have changed to use the endgame-* indices and I don't see a way to change this. This renders the map pretty much useless in our case where the actual external geo data is in the proxy and firewall logs that we ingest and not from endpoints. Can we get an option to customize the data sources or layers for this or at least revert back to normal index geo points?
In 7.6 the Network Map was updated to support pattern matching when creating its configuration, so perhaps what you're seeing is a byproduct of that? While there isn't a dedicated configuration UI (feature request? :), the Map will look for Kibana Index Patterns that match the indices defined in your siem:defaultIndex Kibana Advanced Setting (more details here in the docs), and then create a source/destination/line layer for each. If your maps were working in a previous version and now don't function, I would first double check the docs, and then post back here with a few more details to see if we can't resolve your issue.
Can the Timeline pane be moved to a new TAB, having it render in the same page as the SIEM data you want to investigate makes the interface very clunky and limits the appeal of using timelines, if this was opened in a new tab that could be moved to a second screen would make it much better.
We've been doing some design reviews on Timeline usability and this is definitely an issue we're looking to solve. I'll be sure to reference your feedback, but as with the other items, outlining these in a feature request will be the best way to ensure both Product and Design can reference your feedback.
Can a summary of the ML job anomalies be surfaced on the SIEM Overview page, even the same page that you get from the ML overview page would be fine.
Hi Garrett,
Thanks for the quick and comprehensive reply, looks like you have most of the requests covered already. In any case I create a few feature requests as instructed.
On the Maps question (number 3) you're correct, it was our index sources for SIEM that were the problem, we uses non-standard index names (a hangover from version 6, logs-network-.... logs-endpoint-... etc.) so to make everything work correctly with the SIEM and built-in ML jobs, we use the standard index pattern names as the ILM aliases for these, I had removed some of the custom index names so the at SIEM only picked up the aliases and while that worked for most things, it did not work for the MAP element. Adding the real index patterns back in fixed the map and it now shows the correct layers.
Nice to see the alerting progressing, I reverted to using elastalert as creating a watcher alert with the relevant fields in the message is big task.
I have ended up with 1 alert that looks for severity medium and up.
Some features that are very useful from that are the ability to customise the subject based on data from the alert, ie (severity):(rule name) on (host.name)
That and using query key to only send 1 alert over a period of time for the same host and rule to stop spamming.
Having to create an alert for every rule would be time consuming.
Would love to hear any feedback you have after updating, so keep us posted! And for your previous comment, we're looking to expand the rule alert management to bulk rule actions, so that should improve the UX around having to create an alert for every rule, so stay tuned for some usability enhancements like this in upcoming releases.
I finally had a chance to test the alerting, a little bit of feedback in my testing:
The ability to add fields from the signal other than the predefined {{context.}} fields, i have tried {{context.host.name}} and {{host.name}} could do with an idea on how to reference if possible.
Suppression granularity and control on what suppressing on. IE suppression for a rule and host within x time and further suppression on an exponential scale
The ability to send to an index
And as you have stated, the ability to create actions than run against multiple rules, possibly based on tags as the structure of a Windows alert maybe different than a Network or EDR.
The ability to add fields from the signal other than the predefined {{context.}} fields, i have tried {{context.host.name}} and {{host.name}} could do with an idea on how to reference if possible.
This one is on our short list -- here's the issue for tracking. The hope here is to expose all fields from the source event.
Suppression granularity and control on what suppressing on. IE suppression for a rule and host within x time and further suppression on an exponential scale
Are you speaking of suppressing signals, or alerts? For suppressing signals we're working to add the concept of Exceptions to Rules -- you can follow along here. Not sure if we have anything in the pipeline for suppressing based on volume within x time/some sort of exponential backoff, but I've taken note for a future feature.
The ability to send to an index
Do you mean also archiving the alert to a data index so it can be referenced later/used in visualizations?
Again, thanks for the feedback Philip! This helps us prioritize for future versions
I was refering to suppresing alerts, i can see the option for every hour etc but the option to have them straight away but to suppress rapidly recurring alerts would be useful.
As for send to index, that is correct, building custom visualizations based on the SIEM detections would be very useful.
Here's the issue for snoozing alerts: https://github.com/elastic/kibana/issues/65706
If you could comment on that issue with your interest/use case that'll help the Alerting team prioritize their efforts.
And as for send to index, I have good news for you...
Since detections are just stored in a data index (.siem-signals-[space]), you can actually visualize them right where they're at. The only trick is that you need to check Include system indices when creating a Kibana Index Pattern for it.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
