# Rule Optimization **URL:** \ **Category:** Elastic Security\ **Created:** [January 20, 2025, 8:17am UTC](https://discuss.elastic.co/t/rule-optimization/373387 "2025-01-20T08:17:59Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![SebJen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebjen/32/140768_2.png) [@SebJen](https://discuss.elastic.co/u/SebJen)\ **Post date:** [January 20, 2025, 8:17am UTC](https://discuss.elastic.co/t/rule-optimization/373387/1 "2025-01-20T08:17:59Z")
Hello, I have a general question about designing optimal rules. To my understanding there are three ways of constructing exclusions/filtering : 1. Directly in the query with "NOT" statements 2. Add as a filter 3. Rule exceptions With regards to performance, which one is the most lightweight/optimal, that consumes the least amount of resources? Furthermore, if a KQL has a "NOT" statement in it, does the order matter? Is it in any way beneficial to state the "NOT" statements at the very beginning of a query rather than at the end? In addition, which language is to prefer with regards to performance? KQL, DSL, ESQL, Lucene? Finally, are there any articles/posts/documentation you recommend me reading? ---
**Author:** ![Marshall\_Main](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marshall_main/32/77883_2.png) [@Marshall\_Main](https://discuss.elastic.co/u/Marshall_Main)\ **Post date:** [January 31, 2025, 6:50pm UTC](https://discuss.elastic.co/t/rule-optimization/373387/2 "2025-01-31T18:50:27Z")
Hi @SebJen, Lots of good questions here. In general, different ways of representing the same logic in a rule will have equivalent performance so it comes down to which representation is most convenient for managing the logic. Under the hood, the rule query, filters, and _most_ exceptions are combined into a single DSL query that we send to Elasticsearch. Elasticsearch does query rewriting and optimization as part of the execution process, so I wouldn't expect simple transformations like reordering statements to have much effect. Exceptions do have slower performance if they reference "large" value lists - see [Create and manage value lists | Elastic Security Solution [8.17] | Elastic](https://www.elastic.co/guide/en/security/current/value-lists-exceptions.html#create-value-lists). If the value list referenced by an exception is small, we are able to fetch the values and include them in our initial query to Elasticsearch, but for large value lists we have to make multiple queries instead which can have a significant impact. This is still handy if you have, for example, 100,000 IP addresses that are known-good and want to exclude them from your rules. Regarding languages to use, KQL, DSL, Lucene, and ESQL should all be equivalent performance-wise. The main difference between KQL, DSL, and Lucene is convenience. ESQL provides significant new aggregation and computation capabilities over the other languages and uses a new query engine. Some interesting blog posts with more info on ESQL performance: > **[New to the Elasticsearch repository: ES|QL](https://www.elastic.co/blog/elasticsearch-query-language-esql)** > > The Elasticsearch Query Language (ES|QL) has been added to the Elasticsearch repository. ES|QL is a powerful declarative language that's native to Elasticsearch and designed for composability, express... > **[Announcing Elastic’s piped query language, ES|QL](https://www.elastic.co/blog/esql-elasticsearch-piped-query-language)** > > Introducing ES|QL: Elastic's piped query language. Transform, enrich, and simplify data investigations with concurrent processing, efficient searches across data, and all-in-one screen aggregations an... The benchmarks may be interesting as well: [https://elasticsearch-benchmarks.elastic.co/#tracks/esql/nightly/default/30d](https://elasticsearch-benchmarks.elastic.co/#tracks/esql/nightly/default/30d) Note that the benchmarks where ESQL is outperforming DSL are aggregations, not just simple search queries. I wouldn't expect much benefit from re-implementing any of your existing Query rules into ESQL. The #1 recommendation I have for detection rule performance is ensure that frozen data is not queried by rules. Usually the timestamp on source docs is sufficient for ES to efficiently exclude frozen data, but if the timestamps on incoming docs are not correct then ES can end up querying frozen data repeatedly. See [Configure advanced settings | Elastic Security Solution [8.17] | Elastic](https://www.elastic.co/guide/en/security/current/advanced-settings.html#exclude-cold-frozen-data-rule-executions) for a setting to explicitly exclude frozen data from rules. When writing queries, the most common pitfall I see is using wildcards too much. Wildcards are much more expensive to evaluate than a regular query, and the position of a wildcard within a string also has an impact on query cost. Finally, [EQL sequences](https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-syntax.html) can be tricky with performance. One way performance can be poor with sequences is if the first event in the sequence is very common and later events are much rarer - in that scenario, Elasticsearch must process many candidate sequences. Alternatively, if the first event in the sequence is rare, it starts with a much smaller set of candidates so performance is much better. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 28, 2025, 6:50pm UTC](https://discuss.elastic.co/t/rule-optimization/373387/3 "2025-02-28T18:50:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.