maxClauseCount is set to 1024 error when running "Threat Intel Filebeat Module (v8.x) Indicator Match" rule

Rorb · May 26, 2022, 10:54pm

I'm having some trouble when enabling the "Threat Intel Filebeat Module (v8.x) Indicator Match" rule, it fails with the error:

Bulk Indexing of signals failed: index: ".ds-logs-generic-default-2022.05.26-000010" reason: "maxClauseCount is set to 1024" type: "too_many_clauses" name: "Threat Intel Filebeat Module (v8.x) Indicator Match

The rule is installed by default when the threat intel module is installed in filebeat. Unfortunately I haven't been able to find many other people having trouble with this error in a rule, especially an inbuilt module rule. I'm not sure about the internal running of indicator match rules though I was hoping it would internally manage its queries so these errors do not happen. The error itself is a little strange as my indices.query.bool.max_clause_count value is also set to the default of 4096, though the error is saying it is set to 1024. I have been able to get the rule running only if I turn the indicator index query timestamp filter down to now-1d/d, though obviously that's not ideal. Changing the schedule to run the rule more frequently also did not help.

I'm running ES 8.2.1, Kibana 8.2.1, Filebeat 8.2.1 and my threatintel module is pulling in 4 different collections from anomali.

The definition is:
Index patterns: auditbeat-, endgame-, filebeat-, logs-, packetbeat-, winlogbeat-

Custom query: file.hash.*:* or file.pe.imphash:* or source.ip:* or destination.ip:* or url.full:* or registry.path:*

Rule type: Indicator Match

Timeline template: Generic Threat Match Timeline

Indicator index patterns: filebeat-8*

Indicator mapping: (file.hash.md5 MATCHES threat.indicator.file.hash.md5) OR (file.hash.sha1 MATCHES threat.indicator.file.hash.sha1) OR (file.hash.sha256 MATCHES threat.indicator.file.hash.sha256) OR (file.pe.imphash MATCHES threat.indicator.file.pe.imphash) OR (source.ip MATCHES threat.indicator.ip) OR (destination.ip MATCHES threat.indicator.ip) OR (url.full MATCHES threat.indicator.url.full) OR (registry.path MATCHES threat.indicator.registry.path)

Filters:

event.module: threatintel
event.category: threat
event.kind: enrichment
event.type: indicator

Indicator index query: @timestamp >= "now-30d/d" and event.module:threatintel and (threat.indicator.file.hash.*:* or threat.indicator.file.pe.imphash:* or threat.indicator.ip:* or threat.indicator.registry.path:* or threat.indicator.url.full:*)

Has anyone else had this problem or know how to fix it? I know I can use an enrichment processor which would probably fix the problem, I just didn't think that I was doing anything too special for this out of the box rule to not be working and also don't really want to have to deal with scheduling enrichment policy executions to keep the enrichment policy index up to date if I could somehow get this working.

Rorb · May 30, 2022, 12:37am

Ended up solving this after finding a Github issue in the Elasticsearch repo:

github.com/elastic/elasticsearch

The maximum number of clauses automatically gets set to 1024 when heap size is less than 1GB

opened 12:49PM - 25 Apr 22 UTC

jpountz

>non-issue :Search/Search Team:Search

The code to compute the maximum number of clauses looks like this: ```java … public static int calculateMaxClauseValue(ThreadPool threadPool) { int searchThreadPoolSize = threadPool.info(ThreadPool.Names.SEARCH).getMax(); long heapSize = JvmStats.jvmStats().getMem().getHeapMax().getGb(); return calculateMaxClauseValue(searchThreadPoolSize, heapSize); } static int calculateMaxClauseValue(long threadPoolSize, double heapInGb) { if (threadPoolSize <= 0 || heapInGb <= 0) { return DEFAULT_MAX_CLAUSE_COUNT; } // In a worst-case scenario, each clause may end up using up to 16k of memory // to load postings, positions, offsets, impacts, etc. So we calculate the // maximum number of clauses we can support in a single thread pool by // dividing the heap by 16k (or the equivalent, multiplying the heap in GB by // 64k), and then divide that by the number of possible concurrent search // threads. int maxClauseCount = (int) (heapInGb * 65_536 / threadPoolSize); return Math.max(DEFAULT_MAX_CLAUSE_COUNT, maxClauseCount); } ``` @nkhristinin reported the following surprising behavior. Say you have 1 vCPU and 2 search threads, - If you have 1GB of heap to the JVM then `maxClauseCount` will be 32,768. - But if you give slightly less than 1GB to the heap, e.g. 1023MB, then `long heapSize` will be 0 because of the cast performed by the `getGB()` call, and `maxClauseCount` will be set to 1,024 because ef the `heapInGb <= 0` condition. It might not be a bad thing to be more careful with small heaps, but the code doesn't suggest that this behavior is intentional.

In the issue the author found that having a jvm heap size less that 1GB forced the maximum number of clauses to be set to the default of 1024, which is why even though my node settings said it was 4096, the system forced it to 1024 internally. This was in fact the case for my setup, even though I have assigned more than 1Gb of memory for the ES node the jvm is only assigned half of that.

I believe you can see your max jvm heap size by running: GET _nodes/settings in the dev console of kibana and looking at nodes.*.attributes.ml.max_jvm_size, it seems to be set to half of nodes.*.attributes.ml.machine_memory.

After assigning over 2Gb of memory and waiting for the ES nodes to update the rule ran with no problem.

system · June 27, 2022, 12:37am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.