Ids query takes up to 1 min on cloud elastic

Elastic Stack Elasticsearch
1

Hi Guys,

we are using elasticsearch as managed services. We have ssd disk and 32 gb ram. At the moment we do use only single index with 720 shards. We also use routing in order to point specific shard directly. %99 percent of our queries daily basis, our routing key is like 'dd/mm/yyyy'. I checked data distribution across the shards it is all fine, max shard size is 350 mb, So we know that we do distribute data evenly.

We do not use docvalues.

Memory usage in average %42, cpu is %30, jvm is %40 and avg load is %10.

Index rate per sec is 18, search request per sec is 28. We have less than 1 billion document on elastic.

I can not see anything wrong in the configuration, logs are clear there is no warning or error message.

When it comes to query elastic it is horribly slow, even simple ids query with single id hitting only one shard takes up to 1 minute which is not acceptable.

I am running out of ideas, i checked almost everything but no luck so far.

Do you have any idea what might cause this issue ?

Thanks in advance.

2

Which managed service exactly?

3

This is way too many shards. Since you say that you have a max shard size of 350 MB, your shards are clearly way too small and I think you need at most 10 shards here.

This is probably unnecessary.

I think it's your number of shards and the use of custom routing.