Query super slow whenever first time on an index

Elastic Stack Elasticsearch
1

Similar query

first time: 19.301 s

second time: 0.314 s

index status after the first query

2

cluster status

3

You have 973.385.595 docs with 1.123.353.812.871 (1.1 Tb if I'm not mistaken) with only 2 data nodes?

That might explain that it takes a lot of time to read the data from disk the first time. On the second run, I guess that the FS Cache is playing its role.

What is the query you are running BTW?

Ho! I just saw that BTW:

"spins": "true"

Use SSD Drives!

4 
{

  "size":500,

  "sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],

  "query":

    {"bool":

      {"must":[

        {"query_string":

          {"analyze_wildcard":true,

            "default_field":"message",

            "query":"event_code:5156 AND _type:logs AND exa_parser_name:q-5156 AND host:QRADAR_5156_HOST_NAME AND data_type:process-network AND dest_port:88"

          }

        }],

        "must_not":[]

      }

    }

}
5

Yes 1T data.

It works well on similar physical machines.

6

If you did not notice I updated my answer in the meantime.

7

Thanks,

  1. is there anyway to get the time spend on fetching from disk?
  2. I tried close the index and reopen the index, seems query is still fast. FS cache not clear after close the index?
8
  1. I don't know
  2. Why would it? The OS does not know that we are not going to access the files anymore.