I am creating per day index in Elastic Search ( version: 7.5.1 ). I have 3 nodes in total and 2 shards with one replica each, total disk: 5.6 TB and JVM Heap: 95.8 GB
per day index size is 40 GB with 110m documents.
I am facing a problem that whenever I query on elastic it is taking more than 15 seconds to give response. My first query takes around 12-18 seconds while second query takes only 1-2 seconds and sometimes milliseconds. what could be the exact reason for this behaviour?
secondly, all my shards are getting aligned to single node like some day Node 1 has 2 primary shards or other day Node 3 or Node 2 has. Shards are not getting allocated to different nodes and replica is never assigned. what could be fix here?
Reference query ( email type is keyword )
GET /today_index_20230327/_search
{
"query": {
"bool": {
"filter": [
{
"term": {
"email": {
"value": "abc@gmail.com"
}
}
}
]
}
}
}