Threat Intel and SIEM

[randomguy (random)]

Hi,

I'm looking for a SIEM solution for our company. At the moment I'm trying to do it with ELK.

My goal is:
I would like to create fancy dashboards, alerts with events correlated from Threat Intel.
I would like to update this dashboard and alarm every time a new IOC match was found (retrospective analysis).

I tried to find an official elastic security/SIEM supported way to do it. I thought it is basic SIEM feature so there has to be a solution. But I couldn't find any answer.

My initial idea was to import threat intel to Elasticsearch and correlate events from the output create a special "dynamic" view and use it as a dashboard, and alert source for SIEM detection rules (SQL way of thinking).

Threat feed to Elastic SIEM:
It seems Elastic SIEM can't handle threat intel natively, so I have to install another server with MISP and export feeds from there to ELK SIEM.
I have found ways to do that so it seems doable, I can delete old IOC-s I'm not interested in...
Issues: I isn't an efficient way, and MISP isn't supported and maintained by Elastic.
Threat feed into Elastic: partially checked.

Use detection rules to trigger alerts:
Before version 7.10 it was impossible from SIEM (watcher is not a detection rule...).
Starting from 7.10 there is a slight hope. There is a new feature called "Indicator Match".
As far as I understand it can be used to check if there is a match between IOC and my Log Files, sounds great.
The issue is: "the resulting detection alert does not contain explicit information about which event field(s) match which indicator field(s)".
What I can use is to " include a reference to the field(s) to be matched in the rule Name and rule Description, and ensure that the Timeline template associated with the rule includes pre-defined column(s) for these fields."
So the result will be: receive an alert, have a slight idea where to start digging.
Unfortunately I can't find any way to create events form detection rules and highlight a single log entry.
As far as I see correlation with Elastic is a nightmare.
I can't create a KQL to correlate two fields from different indexes.
I can't create Query DSL to correlate two fields from different indexes.
I can't create an SQL like query which include join and sub query-s to correlate two fields from different indexes.
If I receive an alert I have to export the affected logs and manually correlate them with IOC-s by custom-made scripts or feed it to an SQL DB... It isn't an efficient way...
Alerts from SIEM: there is a half-baked solution, it requires a lot of manual work in order to correlate.

Create a dashboard from the correlated events (and it has to be dynamically updated with new IOC-s...):
The only way I found is enrich the data before indexing unfortunately by doing that I lose the retrospective analysis. Ouch.
It can be done by Logstash or by Enrich processor.
The logstash way: a lot of people try to use memcached in order to enrich data and index high volume of events.
The issues with this: I have to import IOC-s to Elastic, then I import IOC-s to memcached with a different tool and consume more resources, memcached itself isn't supported by elastic (there is a plugin but they don't support the product itself). It doubles the maintenance costs.
It seems to me that Elastic is focused on Beats... not on Logstash, I read Logstash isn't efficient and I should minimize the usage of it and focus on the new way: Beats.

I thought I can use Beats to enrich data. Unfortunately I can't find any way to do that.
For example I would like to use Filebeat with netflow module and enrich the data with IOC-s form MISP by memcached?
The solution I found is to send beats output to Logstash and re parse the beats output and enrich it.
It seems like there is a contradiction. Try to use Beats it is light and efficient ohh wait use it with Logstas to enrich...

The next idea was to use the enrich processor.
Read "The approach of keeping threat intel in ES and checking against it for every ingested log entry is quite heavy". So it seems to be another dead end.

I spent 1 month to figure out how to deal with Threat Intel in Elastic SIEM.
Before I give up.

Is there an efficient and easy way to use Threat Intel with Elastic SIEM built in tools?
If there is please send me a link.

[Mike_Paquette]

Heya @randomguy, welcome to our community!

It is clear that you've invested significant time checking out and using Elastic SIEM/Security. As the team that develops this solution, we sincerely appreciate the feedback that you've provided, and we thank you for taking the time to write it all down! Please let me address some of your specific observations and questions.

Last question first:

Is there an efficient and easy way to use Threat Intel with Elastic SIEM built in tools?
If there is please send me a link.

We're on a fast-moving journey to build additional capabilities into our security solution, and threat intelligence integration is coming into focus right now, but we're not there yet. The timing of your input is ideal, so we hope you'll remain engaged with our community as they roll out.

Threat feed to Elastic SIEM:

• It seems Elastic SIEM can't handle threat intel natively, so I have to install another server with MISP and export feeds from there to ELK SIEM.
• I have found ways to do that so it seems doable, I can delete old IOC-s I'm not interested in...
• Issues: I isn't an efficient way, and MISP isn't supported and maintained by Elastic.
• Threat feed into Elastic: partially checked.

It is true that in the 7.10 release, it requires manual effort to ingest threat intelligence feed data (aka IOC’s) into Elasticsearch indices, so they can be used as part of correlation with detection events. As you have found, there are several ways to do this, including via Beats, Logstash, the “Import Data” function of the machine learning app, and, even SIEM/Security Value lists, which are normally used for exceptions.

Can you provide us with some of the threat intelligence feeds that you think are valuable, and for which you’d like to see a more automated ingestion and management process?

We are currently working on a future experience in the security-app, where you’ll be able to select and configure the threat intel feeds that you want to use, and get the data ingested into indicator indices for use in your correlations.

Use detection rules to trigger alerts:

• Before version 7.10 it was impossible from SIEM (watcher is not a detection rule...).
• Starting from 7.10 there is a slight hope. There is a new feature called "Indicator Match".
• As far as I understand it can be used to check if there is a match between IOC and my Log Files, sounds great.
• The issue is: "the resulting detection alert does not contain explicit information about which event field(s) match which indicator field(s)".
• What I can use is to " include a reference to the field(s) to be matched in the rule Name and rule Description, and ensure that the Timeline template associated with the rule includes pre-defined column(s) for these fields."
• So the result will be: receive an alert, have a slight idea where to start digging.

Yes, the Indicator match rule type is how the security app provides for checking for matches between IOC and your log files. You are correct that in the 7.10 release, the rule type does not provide the important details about which event field(s) match which indicator field(s). New functionality (2nd phase) is being actively developed that will provide this information in the detection alerts (documents in the .siem-signals-<space> indices).

Unfortunately I can't find any way to create events form detection rules and highlight a single log entry.

Not sure I understand this comment fully, but for SIEM/Security detection rule types that identify a unique source log event, the Elasticsearch _id of the event is stored in the signal.parent.id field, which lets you find the single log entry, shown here in the Timeline view:

As far as I see correlation with Elastic is a nightmare.

• I can't create a KQL to correlate two fields from different indexes.
• I can't create Query DSL to correlate two fields from different indexes.
• I can't create an SQL like query which include join and sub query-s to correlate two fields from different indexes.
• If I receive an alert I have to export the affected logs and manually correlate them with IOC-s by custom-made scripts or feed it to an SQL DB... It isn't an efficient way...
• Alerts from SIEM: there is a half-baked solution, it requires a lot of manual work in order to correlate.

Again, we are on a fast-moving journey adding correlation capabilities to the Elastic Stack and the the security solution. Our new Event Query Language (EQL) was introduced into the Elastic Stack in the 7.9 release, and the Security App in 7.10 introduced the new "Event Correlation" rule type that allows an EQL query to be specified. In the future, we expect that EQL will become available in more places throughout Kibana, allowing flexible use if correlation throughout the analyst experience.

Create a dashboard from the correlated events (and it has to be dynamically updated with new IOC-s...):

*The only way I found is enrich the data before indexing unfortunately by doing that I lose the retrospective analysis. Ouch.

Users are not required to use ingestion-time enrichment to accomplish a dashboard of log events correlated with threat intelligence indicators.

You can easily create visualizations from Kibana Discover, Kibana Dashboards or even Canvas today that list all correlations identified by the Indicator match rules, by simply specifying the .siem-signals-<space> index and filtering for your specific indicator match rules. Here's a Lens-based visualization based on the .siem-signals-* index pattern, showing the indicator-match. rules included in this index pattern.

Thanks again!

[randomguy (random)]

Hello Mike,

Thanks for the detailed answer. I believe you are on the right track, since I started to learn more about your product I have seen an insane fast development. EQL is also a huge step to improve the detection capabilities, but there is still room to improve the product further. I'm looking forward for the improved version of the SIEM.

Can you provide us with some of the threat intelligence feeds that you think are valuable, and for which you’d like to see a more automated ingestion and management process?

I would like to be able to import custom feeds not to limit my options. Simple CSV Parsed Feed, Text feed, or STIX/TAXII feeds. But it is easier sad than done (some vendors are open some require authentication... like Alienvault, AbuseIPDB, IBM X-Force ), deduplication dealing with false positives, deleting old entries, combine it with ECS, skipping custom fields, proxy support (within an enterprise network usually it is mandatory). It isn't an easy job...

It would be also nice to be able to easily create thing like objects and object groups for example: I would like to create a detection rule to detect port scan attempt initiated form the users network.
Or detect events from privileged users, special rules for server network or DMZ network, there are many use case for the objects. At the moment you can create manually hardcoded rules or enrich the data, I believe maintaining hardcoded rules is a nightmare. With objects it is more efficient and flexible.