How can I identify the root cause and fix a query in Elasticsearch cluster that never returns a response

How can I identify the root cause and fix a query in Elasticsearch cluster that never returns a response.

I have an Elasticsearch cluster with Elasticsearch, Logstash, and Kibana running on the same machine using Docker compose. The Elastic version is 8.3, and the cluster is used for logging. The index lifecycle management policy is set to rollover the index if it exceeds 40 GB in size, and the log retention period is set to one day. The machine has 2 VCPU and 8 GB RAM, and a 200 GB gp3 disk.

While all queries work fast, one particular query never returns a response, especially when searching logs for application B combined with some text in the message field. The query is shown below and works fast for application A but not for application B:
One thing that I know is that the message field for application A are very large compared to my other applications.

{
  "track_total_hits": false,
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "fields": [
    {
      "field": "*",
      "include_unmapped": "true"
    },
    {
      "field": "@timestamp",
      "format": "strict_date_optional_time"
    }
  ],
  "size": 500,
  "version": true,
  "script_fields": {},
  "stored_fields": [
    "*"
  ],
  "runtime_mappings": {},
  "_source": false,
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "filter": [
              {
                "bool": {
                  "should": [
                    {
                      "match_phrase": {
                        "message": "app is running"
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              },
              {
                "bool": {
                  "should": [
                    {
                      "term": {
                        "app_name": "A"
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              }
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2023-04-05T11:15:16.781Z",
              "lte": "2023-04-05T11:30:16.781Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  },
  "highlight": {
    "pre_tags": [
      "@kibana-highlighted-field@"
    ],
    "post_tags": [
      "@/kibana-highlighted-field@"
    ],
    "fields": {
      "*": {}
    },
    "fragment_size": 2147483647
  }
}

The query fails on a very small time frame also.
How do I identify the root cause and fix the issue ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.