# maxClauseCount is set to 1024 error when running "Threat Intel Filebeat Module (v8.x) Indicator Match" rule **URL:** \ **Category:** Elastic Security\ **Tags:** detection-rules\ **Created:** [May 26, 2022, 10:54pm UTC](https://discuss.elastic.co/t/maxclausecount-is-set-to-1024-error-when-running-threat-intel-filebeat-module-v8-x-indicator-match-rule/305757 "2022-05-26T22:54:07Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Rorb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rorb/32/98764_2.png) [@Rorb](https://discuss.elastic.co/u/Rorb)\ **Post date:** [May 26, 2022, 10:54pm UTC](https://discuss.elastic.co/t/maxclausecount-is-set-to-1024-error-when-running-threat-intel-filebeat-module-v8-x-indicator-match-rule/305757/1 "2022-05-26T22:54:07Z")
I'm having some trouble when enabling the "Threat Intel Filebeat Module (v8.x) Indicator Match" rule, it fails with the error: > Bulk Indexing of signals failed: index: ".ds-logs-generic-default-2022.05.26-000010" reason: "maxClauseCount is set to 1024" type: "too\_many\_clauses" name: "Threat Intel Filebeat Module (v8.x) Indicator Match The rule is installed by default when the threat intel module is installed in filebeat. Unfortunately I haven't been able to find many other people having trouble with this error in a rule, especially an inbuilt module rule. I'm not sure about the internal running of indicator match rules though I was hoping it would internally manage its queries so these errors do not happen. The error itself is a little strange as my `indices.query.bool.max_clause_count` value is also set to the default of 4096, though the error is saying it is set to 1024. I have been able to get the rule running only if I turn the indicator index query timestamp filter down to `now-1d/d`, though obviously that's not ideal. Changing the schedule to run the rule more frequently also did not help. I'm running ES 8.2.1, Kibana 8.2.1, Filebeat 8.2.1 and my threatintel module is pulling in 4 different collections from anomali. The definition is: **Index patterns:** auditbeat-_, endgame-_, filebeat-_, logs-_, packetbeat-_, winlogbeat-_ **Custom query:** `file.hash.*:* or file.pe.imphash:* or source.ip:* or destination.ip:* or url.full:* or registry.path:*` **Rule type:** Indicator Match **Timeline template:** Generic Threat Match Timeline **Indicator index patterns:** filebeat-8\* **Indicator mapping:** `(file.hash.md5 MATCHES threat.indicator.file.hash.md5) OR (file.hash.sha1 MATCHES threat.indicator.file.hash.sha1) OR (file.hash.sha256 MATCHES threat.indicator.file.hash.sha256) OR (file.pe.imphash MATCHES threat.indicator.file.pe.imphash) OR (source.ip MATCHES threat.indicator.ip) OR (destination.ip MATCHES threat.indicator.ip) OR (url.full MATCHES threat.indicator.url.full) OR (registry.path MATCHES threat.indicator.registry.path)` **Filters:** - event.module: threatintel - event.category: threat - event.kind: enrichment - event.type: indicator **Indicator index query:** `@timestamp >= "now-30d/d" and event.module:threatintel and (threat.indicator.file.hash.*:* or threat.indicator.file.pe.imphash:* or threat.indicator.ip:* or threat.indicator.registry.path:* or threat.indicator.url.full:*)` Has anyone else had this problem or know how to fix it? I know I can use an enrichment processor which would probably fix the problem, I just didn't think that I was doing anything too special for this out of the box rule to not be working and also don't really want to have to deal with scheduling enrichment policy executions to keep the enrichment policy index up to date if I could somehow get this working. ---
**Author:** ![Rorb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rorb/32/98764_2.png) [@Rorb](https://discuss.elastic.co/u/Rorb)\ **Post date:** [May 30, 2022, 12:37am UTC](https://discuss.elastic.co/t/maxclausecount-is-set-to-1024-error-when-running-threat-intel-filebeat-module-v8-x-indicator-match-rule/305757/2 "2022-05-30T00:37:20Z")
Ended up solving this after finding a Github issue in the Elasticsearch repo: > > > The code to compute the maximum number of clauses looks like this: > > \`\`\`java > … public static int calculateMaxClauseValue(ThreadPool threadPool) { > int searchThreadPoolSize = threadPool.info(ThreadPool.Names.SEARCH).getMax(); > long heapSize = JvmStats.jvmStats().getMem().getHeapMax().getGb(); > return calculateMaxClauseValue(searchThreadPoolSize, heapSize); > } > > static int calculateMaxClauseValue(long threadPoolSize, double heapInGb) { > if (threadPoolSize \<= 0 || heapInGb \<= 0) { > return DEFAULT\_MAX\_CLAUSE\_COUNT; > } > // In a worst-case scenario, each clause may end up using up to 16k of memory > // to load postings, positions, offsets, impacts, etc. So we calculate the > // maximum number of clauses we can support in a single thread pool by > // dividing the heap by 16k (or the equivalent, multiplying the heap in GB by > // 64k), and then divide that by the number of possible concurrent search > // threads. > int maxClauseCount = (int) (heapInGb \* 65\_536 / threadPoolSize); > return Math.max(DEFAULT\_MAX\_CLAUSE\_COUNT, maxClauseCount); > } > \`\`\` > > @nkhristinin reported the following surprising behavior. Say you have 1 vCPU and 2 search threads, > - If you have 1GB of heap to the JVM then \`maxClauseCount\` will be 32,768. > - But if you give slightly less than 1GB to the heap, e.g. 1023MB, then \`long heapSize\` will be 0 because of the cast performed by the \`getGB()\` call, and \`maxClauseCount\` will be set to 1,024 because ef the \`heapInGb \<= 0\` condition. > > It might not be a bad thing to be more careful with small heaps, but the code doesn't suggest that this behavior is intentional. In the issue the author found that having a jvm heap size less that 1GB forced the maximum number of clauses to be set to the default of 1024, which is why even though my node settings said it was 4096, the system forced it to 1024 internally. This was in fact the case for my setup, even though I have assigned more than 1Gb of memory for the ES node the jvm is only assigned half of that. I believe you can see your max jvm heap size by running: `GET _nodes/settings` in the dev console of kibana and looking at `nodes.*.attributes.ml.max_jvm_size`, it seems to be set to half of `nodes.*.attributes.ml.machine_memory`. After assigning over 2Gb of memory and waiting for the ES nodes to update the rule ran with no problem. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 27, 2022, 12:37am UTC](https://discuss.elastic.co/t/maxclausecount-is-set-to-1024-error-when-running-threat-intel-filebeat-module-v8-x-indicator-match-rule/305757/3 "2022-06-27T00:37:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.