Indicator Match Rule Fails with too_many_nested_clauses

Elastic Security SIEM
1

Hi,

I created an indicator match rule using intel from the Recorded Future integration package v1.0.1

I am getting the following failure when the rule runs:

Bulk Indexing of signals failed: ResponseError: search_phase_execution_exception: [too_many_nested_clauses] Reason: Query contains too many nested clauses; maxClauseCount is set to 37449 name: "URL detections from Recorded Future" id: "7078a5c0-fe4b-11ec-a2d8-071a4b73939e" rule id: "f010fb0e-77e2-4fbd-bc46-6aeac58bbc0c" execution id: "b2362ee5-b44e-4823-b5c3-131a64933d37" space ID: "default"

The rule succeeds occasionally, but also does not find matches when I know matches exist. Here is the rule logic, and the same indicator showing in both my index and the logs-ti_* datastream:

Screen Shot 2022-07-08 at 4.26.35 PM
Screen Shot 2022-07-08 at 4.26.35 PM1409×598 65.6 KB

Screen Shot 2022-07-08 at 4.29.39 PM
Screen Shot 2022-07-08 at 4.29.39 PM1486×774 138 KB

Screen Shot 2022-07-08 at 4.29.51 PM
Screen Shot 2022-07-08 at 4.29.51 PM1298×819 151 KB

2

Hey, what version of Kibana do you have?

Can I also see mapping for demo_url? url.full in particular

About that, you don't have alerts:
In your indicator index query you have `@timestamp >= "now-30d/d". Does this URL from logs-ti_* fit this time query?

3

Hey Nikita,

Kibana v8.1.2

url.full mapping:

        "url" : {
          "properties" : {
            "full" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        }

About that, you don't have alerts:
In your indicator index query you have `@timestamp >= "now-30d/d". Does this URL from logs-ti_* fit this time query?

The timeframe "now-30d/d" gets a lot of documents from logs-ti_*, was hoping if I used more -> more chance to make alert. I think this was pre-populated creating the indicator match rule. I could change it if that is better. I only need the most recent intel, the demo_url index intentionally has a lot of same indicators as logs-ti_*

Over 10k hits on this query (I added the url tag in integration settings):

GET /logs-ti_*/_search
{
  "query": {
    "bool": {
      "must": [
        {"range": {"@timestamp": {"gte": "now-30d/d"}}},
        {"term": {"tags": {"value": "url"}}}
      ]
    }
  }
}

demo_url full mapping:

"demo_url" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "event" : {
          "properties" : {
            "original" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "host" : {
          "properties" : {
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "http_request_method" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "http_response_status_code" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "log" : {
          "properties" : {
            "file" : {
              "properties" : {
                "path" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            }
          }
        },
        "message" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "mime_content_type" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "reply_size_include_header" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "response_time" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "server_ip" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "source_ip" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "squid_hierarchy_status" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "squid_request_status" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "tags" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "timestamp" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "url" : {
          "properties" : {
            "full" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "user" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
4

I believe one of the reasons you have an error in the first place is because there url.full has a mapping text. It performs a full-text search then and creates queries with too many clauses.

What you can do, it's change the Indicator mapping in the rule from url.full to url.full.keyword, disable and enable the rule again. I hope the error will go and you can see some alerts

If after that, you don't have the error, but also don't have alerts, maybe it's because the match doesn't happen. You can try to change Indicator index query maybe to a bigger interval. But remember that Indicator Match rule has some performance limitation, this is why we have this query by default. (you can also upgrade to 8.3 it's has some improvements in the performance)

1 Like
5

Thank you @Nikita_Khristinin - I am using url.full.keyword now and the alert rule runs are succeeding now. I think we can close this now. I am looking into the lack of matches further and I believe that part is just my data

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.