Threat Intel filebeat module

[SiRVaNdO (Servando)]

Hello everyone,
I installed a filebeat with the threat intel module and it's importing threat intel data to the Elasticsearch. When I visit the feeds dashboards all is working properly.

The problem comes when I tried to use those feeds . For example, if I use in the discover the Index pattern I get this error:
"Unable to retrieve the requested [fields] since _source is disabled in the mappings for index [filebeat-7.14.1-2021.09.12-000001]"

And If I tried to use the data in an Indicator Match rule I get this error:
An error occurred during rule execution: message: "search_phase_execution_exception: [illegal_argument_exception] Reason: Unable to retrieve the requested [fields] since _source is disabled in the mappings for index [filebeat-7.14.1-2021.09.12-000001]" name: "Threat Intelligence IP match" id: "3a75d530-13b2-11ec-89ba-6daa40144b8a" rule id: "e8cfdfbf-6b57-4749-98b9-5073ae573cfd" signals index: ".siem-signals-default"

All the index mapping, metadata & information is generated automatically by filebeat in the setup with the command filebeat -e setup
What am I doing wrong?

Thanks in advance,

V.

[SiRVaNdO (Servando)]

I have progress. I deactivate ILM by putting this in the filebeat.yml config file:

setup.ilm.enabled: false

After that, filebeat ignores this config:

Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

2021-09-12T17:23:59.168+0200	INFO	[index-management]	idxmgmt/std.go:261	Auto ILM enable success.
2021-09-12T17:23:59.178+0200	INFO	[index-management.ilm]	ilm/std.go:160	ILM policy filebeat exists already.
2021-09-12T17:23:59.178+0200	INFO	[index-management]	idxmgmt/std.go:401	Set setup.template.name to '{filebeat-7.14.1 {now/d}-000001}' as ILM is enabled.
2021-09-12T17:23:59.178+0200	INFO	[index-management]	idxmgmt/std.go:406	Set setup.template.pattern to 'filebeat-7.14.1-*' as ILM is enabled.
2021-09-12T17:23:59.178+0200	INFO	[index-management]	idxmgmt/std.go:440	Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.14.1 {now/d}-000001} as ILM is enabled.
2021-09-12T17:23:59.178+0200	INFO	[index-management]	idxmgmt/std.go:444	Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.

But I have a different behavior. Now I can navigate through discover threat intel data. The problem is when I do an Indicator match rule. I get this error:

An error occurred during rule execution: message: "Request timed out" name: "Threat Intelligence IP match" id: "3a75d530-13b2-11ec-89ba-6daa40144b8a" rule id: "e8cfdfbf-6b57-4749-98b9-5073ae573cfd" signals index: ".siem-signals-default"

The default Threat Intel rule, defined by filebeat setup, is also failing with the same error.

I tried to do the match with an index pattern with 1k entries and is failing also by timeout. I think that there is something wrong with this feeds.

Any clue?

[n2x4 (Nate)]

I had the same issue with the default rule timing out. I think it's simply too complex of a query for some environments and needs to be broken down into multiple rules.

I had success by copying the rule, and then removing everything except IP related fields (remove hashes and domains). Then save your new rule as something like "IP Threat Intel".

[SiRVaNdO (Servando)]

I tried that and fails with the same error.
I tried also with indexes with a few entries and get the same error.
I'm trying now to feed threat intel manually and check if it is the indicator match rule or the filebeat feeds.

I Keep investigating,

[SiRVaNdO (Servando)]

I'm stucked in that problem. I don't have an indicator mach rule working properly. Anyone knows a blog or documentation to review?

[samiusan (Michael)]

Hi,

I am having a similar issue (Request timed out when running Threat Intel match rule).

I have been trying to limit the match rules alert rules to minimal information (i.e. only IP address information) without luck.

Wondering if anyone has had any more luck making this work?

Also how can you run a match query from the command line? I'm trying to check the query itself works correctly and see what other options are available to tune it.

cheers

[SiRVaNdO (Servando)]

Hi there,
I found a solution. Like @n2x4 said, you have to coy the rule and divide it.
Also I increased the timeouts in kibana/elasticsearch in order to execute those queries properly.

Thanks!!!

[PublicName]

Setting the time to search every 60 minutes and removing the Endgame index and packetbeat works wornder's. No more failures. I agree it's a very complex query with how much it looks for in one pass.

Do you happen to end up with:

or do you actually get data populating in the enrichment part? I've been over that setup a dozen time and it just seems odd I get alerts and can get the IP or hash but the enrichment is empty.

[syunusic (Sacha Yunusic)]

I've exactly this problem. I have a very simple rule with only 20,000 entries in filebeat-* and only 50,000 in the other index (It's a Firewall traffic index). Every time it runs, I see new entries in .siem-signals-default-*, but the same timeout error than you. Every time.
It doesn't trigger the action (send email or whatever you configure there) at all.

[MichaelHuff (Michael Lykins)]

I had to completely disable the threat intel feed in my cluster as it was searching and actively hitting on a couple items, but ultimately the performance hit was way to massive. Have not been running it for a couple months now.

I hope they get it fixed, it was very useful.