# Security Detection Rules Cause: \`circuit\_breaking\_exception\` on medium-ish deployments **URL:** \ **Category:** SIEM\ **Tags:** detection-rules\ **Created:** [October 13, 2021, 6:43pm UTC](https://discuss.elastic.co/t/security-detection-rules-cause-circuit-breaking-exception-on-medium-ish-deployments/286654 "2021-10-13T18:43:09Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [October 13, 2021, 6:43pm UTC](https://discuss.elastic.co/t/security-detection-rules-cause-circuit-breaking-exception-on-medium-ish-deployments/286654/1 "2021-10-13T18:43:10Z")
Hi All, I noticed that a few detection rules consume a lot of memory, and cause `circuit_breaking_exception` often in medium-ish deployments (~265 winlogbeat deployments). Elasticsearch version: `7.14.1` Offending Rules: - Installation of Custom Shim Databases - Parent Process PID Spoofing - Potential Process Herpaderping Attempt These rules all seem to use a good amount of memory to run, here is generally the exception I see: ```auto An error occurred during rule execution: message: "circuit_breaking_exception: [circuit_breaking_exception] Reason: [eql_sequence] Data too large, data for [sequence_inflight] would be [3221924600/3gb], which is larger than the limit of [3221225472/3gb]" name: "Installation of Custom Shim Databases" id: "ac10bafe-a91f-11eb-a252-7f35a8822039" rule id: "c5ce48a6-7f57-4ee8-9313-3d0024caee10" signals index: ".siem-signals-security" ``` Has anyone else run into this issue with rules, if you did, how did you solve it? ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [October 13, 2021, 6:46pm UTC](https://discuss.elastic.co/t/security-detection-rules-cause-circuit-breaking-exception-on-medium-ish-deployments/286654/2 "2021-10-13T18:46:27Z")
Further context, the rules are executed across 3 coordinating only nodes in the cluster each with 8GB of RAM (6GB of Heap). ---
**Author:** ![RylandHerrick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rylandherrick/32/67401_2.png) [@RylandHerrick](https://discuss.elastic.co/u/RylandHerrick)\ **Post date:** [October 14, 2021, 11:44pm UTC](https://discuss.elastic.co/t/security-detection-rules-cause-circuit-breaking-exception-on-medium-ish-deployments/286654/3 "2021-10-14T23:44:57Z")
Hi @BenB196! I can't say for certain, but since all of those rules are EQL rules, I'm guessing that you may be missing fields that the EQL sequence is attempting to join upon. It's likely `process.entity_id`, but you'd have to verify that. What EQL does in the case of a missing join field **is to treat them as null and join on that value** , which results in a lot of unexpected results/sequences, and causes the circuit breaking exception you're seeing. While we're actively discussing a fix for this behavior, the near term solution would be to duplicate those rules and edit the sequence to either remove that join field, or substitute it for a more appropriate one for your data. I hope that helps, cheers! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 15, 2021, 12:11am UTC](https://discuss.elastic.co/t/security-detection-rules-cause-circuit-breaking-exception-on-medium-ish-deployments/286654/4 "2021-10-15T00:11:24Z")
What is the output from the `_cluster/stats?pretty&human` API in Elasticsearch? ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [October 18, 2021, 1:50pm UTC](https://discuss.elastic.co/t/security-detection-rules-cause-circuit-breaking-exception-on-medium-ish-deployments/286654/5 "2021-10-18T13:50:40Z")
Here is the output of the requested command: [{ "\_nodes" : { "total" : 30, "successful" : 30, "failed" : 0 - Pastebin.com](https://pastebin.com/Y7fczXaK) **Note** : I've upgraded the cluster to 7.15.1 since originally making this post. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 19, 2021, 2:55am UTC](https://discuss.elastic.co/t/security-detection-rules-cause-circuit-breaking-exception-on-medium-ish-deployments/286654/6 "2021-10-19T02:55:07Z")
Thanks for that. What size heaps do you run, it looks like ~6GB per node? ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [October 19, 2021, 12:51pm UTC](https://discuss.elastic.co/t/security-detection-rules-cause-circuit-breaking-exception-on-medium-ish-deployments/286654/7 "2021-10-19T12:51:53Z")
Yes, the coord nodes run ~6GB (they have 8GB total, and use the auto heap setting so whatever the Elasticsearch decides for their heap is what they get). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 16, 2021, 12:52pm UTC](https://discuss.elastic.co/t/security-detection-rules-cause-circuit-breaking-exception-on-medium-ish-deployments/286654/8 "2021-11-16T12:52:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.