maxClauseCount is set to 1024 error when running "Threat Intel Filebeat Module (v8.x) Indicator Match" rule

Elastic Security
1

I'm having some trouble when enabling the "Threat Intel Filebeat Module (v8.x) Indicator Match" rule, it fails with the error:

Bulk Indexing of signals failed: index: ".ds-logs-generic-default-2022.05.26-000010" reason: "maxClauseCount is set to 1024" type: "too_many_clauses" name: "Threat Intel Filebeat Module (v8.x) Indicator Match

The rule is installed by default when the threat intel module is installed in filebeat. Unfortunately I haven't been able to find many other people having trouble with this error in a rule, especially an inbuilt module rule. I'm not sure about the internal running of indicator match rules though I was hoping it would internally manage its queries so these errors do not happen. The error itself is a little strange as my indices.query.bool.max_clause_count value is also set to the default of 4096, though the error is saying it is set to 1024. I have been able to get the rule running only if I turn the indicator index query timestamp filter down to now-1d/d, though obviously that's not ideal. Changing the schedule to run the rule more frequently also did not help.

I'm running ES 8.2.1, Kibana 8.2.1, Filebeat 8.2.1 and my threatintel module is pulling in 4 different collections from anomali.

The definition is:
Index patterns: auditbeat-, endgame-, filebeat-, logs-, packetbeat-, winlogbeat-

Custom query: file.hash.*:* or file.pe.imphash:* or source.ip:* or destination.ip:* or url.full:* or registry.path:*

Rule type: Indicator Match

Timeline template: Generic Threat Match Timeline

Indicator index patterns: filebeat-8*

Indicator mapping: (file.hash.md5 MATCHES threat.indicator.file.hash.md5) OR (file.hash.sha1 MATCHES threat.indicator.file.hash.sha1) OR (file.hash.sha256 MATCHES threat.indicator.file.hash.sha256) OR (file.pe.imphash MATCHES threat.indicator.file.pe.imphash) OR (source.ip MATCHES threat.indicator.ip) OR (destination.ip MATCHES threat.indicator.ip) OR (url.full MATCHES threat.indicator.url.full) OR (registry.path MATCHES threat.indicator.registry.path)

Filters:

  • event.module: threatintel
  • event.category: threat
  • event.kind: enrichment
  • event.type: indicator

Indicator index query: @timestamp >= "now-30d/d" and event.module:threatintel and (threat.indicator.file.hash.*:* or threat.indicator.file.pe.imphash:* or threat.indicator.ip:* or threat.indicator.registry.path:* or threat.indicator.url.full:*)

Has anyone else had this problem or know how to fix it? I know I can use an enrichment processor which would probably fix the problem, I just didn't think that I was doing anything too special for this out of the box rule to not be working and also don't really want to have to deal with scheduling enrichment policy executions to keep the enrichment policy index up to date if I could somehow get this working.

2

Ended up solving this after finding a Github issue in the Elasticsearch repo:

In the issue the author found that having a jvm heap size less that 1GB forced the maximum number of clauses to be set to the default of 1024, which is why even though my node settings said it was 4096, the system forced it to 1024 internally. This was in fact the case for my setup, even though I have assigned more than 1Gb of memory for the ES node the jvm is only assigned half of that.

I believe you can see your max jvm heap size by running: GET _nodes/settings in the dev console of kibana and looking at nodes.*.attributes.ml.max_jvm_size, it seems to be set to half of nodes.*.attributes.ml.machine_memory.

After assigning over 2Gb of memory and waiting for the ES nodes to update the rule ran with no problem.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.