Threat Intel filebeat module

Hello everyone,
I installed a filebeat with the threat intel module and it's importing threat intel data to the Elasticsearch. When I visit the feeds dashboards all is working properly.

The problem comes when I tried to use those feeds . For example, if I use in the discover the Index pattern I get this error:
"Unable to retrieve the requested [fields] since _source is disabled in the mappings for index [filebeat-7.14.1-2021.09.12-000001]"

And If I tried to use the data in an Indicator Match rule I get this error:
An error occurred during rule execution: message: "search_phase_execution_exception: [illegal_argument_exception] Reason: Unable to retrieve the requested [fields] since _source is disabled in the mappings for index [filebeat-7.14.1-2021.09.12-000001]" name: "Threat Intelligence IP match" id: "3a75d530-13b2-11ec-89ba-6daa40144b8a" rule id: "e8cfdfbf-6b57-4749-98b9-5073ae573cfd" signals index: ".siem-signals-default"

All the index mapping, metadata & information is generated automatically by filebeat in the setup with the command filebeat -e setup
What am I doing wrong?

Thanks in advance,

V.

I have progress. I deactivate ILM by putting this in the filebeat.yml config file:

setup.ilm.enabled: false

After that, filebeat ignores this config:

Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

2021-09-12T17:23:59.168+0200	INFO	[index-management]	idxmgmt/std.go:261	Auto ILM enable success.
2021-09-12T17:23:59.178+0200	INFO	[index-management.ilm]	ilm/std.go:160	ILM policy filebeat exists already.
2021-09-12T17:23:59.178+0200	INFO	[index-management]	idxmgmt/std.go:401	Set setup.template.name to '{filebeat-7.14.1 {now/d}-000001}' as ILM is enabled.
2021-09-12T17:23:59.178+0200	INFO	[index-management]	idxmgmt/std.go:406	Set setup.template.pattern to 'filebeat-7.14.1-*' as ILM is enabled.
2021-09-12T17:23:59.178+0200	INFO	[index-management]	idxmgmt/std.go:440	Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.14.1 {now/d}-000001} as ILM is enabled.
2021-09-12T17:23:59.178+0200	INFO	[index-management]	idxmgmt/std.go:444	Set settings.index.lifecycle.name in template to {filebeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.

But I have a different behavior. Now I can navigate through discover threat intel data. The problem is when I do an Indicator match rule. I get this error:

An error occurred during rule execution: message: "Request timed out" name: "Threat Intelligence IP match" id: "3a75d530-13b2-11ec-89ba-6daa40144b8a" rule id: "e8cfdfbf-6b57-4749-98b9-5073ae573cfd" signals index: ".siem-signals-default"

The default Threat Intel rule, defined by filebeat setup, is also failing with the same error.

I tried to do the match with an index pattern with 1k entries and is failing also by timeout. I think that there is something wrong with this feeds.

Any clue?

I had the same issue with the default rule timing out. I think it's simply too complex of a query for some environments and needs to be broken down into multiple rules.

I had success by copying the rule, and then removing everything except IP related fields (remove hashes and domains). Then save your new rule as something like "IP Threat Intel".

I tried that and fails with the same error.
I tried also with indexes with a few entries and get the same error.
I'm trying now to feed threat intel manually and check if it is the indicator match rule or the filebeat feeds.

I Keep investigating,

I'm stucked in that problem. I don't have an indicator mach rule working properly. Anyone knows a blog or documentation to review?

Hi,

I am having a similar issue (Request timed out when running Threat Intel match rule).

I have been trying to limit the match rules alert rules to minimal information (i.e. only IP address information) without luck.

Wondering if anyone has had any more luck making this work?

Also how can you run a match query from the command line? I'm trying to check the query itself works correctly and see what other options are available to tune it.

cheers

Hi there,
I found a solution. Like @n2x4 said, you have to coy the rule and divide it.
Also I increased the timeouts in kibana/elasticsearch in order to execute those queries properly.

Thanks!!!

Setting the time to search every 60 minutes and removing the Endgame index and packetbeat works wornder's. No more failures. I agree it's a very complex query with how much it looks for in one pass.

Do you happen to end up with:

or do you actually get data populating in the enrichment part? I've been over that setup a dozen time and it just seems odd I get alerts and can get the IP or hash but the enrichment is empty.