High Search Rate spikes for AWS elasticsearch

[mehak]

Hi Team,

We are seeing high SearchRate spike at a certain times in a day along with Index and Search latency. We are having a cluster of 6 nodes with 3 master ones and other 3 are data nodes with 200 GB EBS volume. Not able to figure out the exact reason for sudden increase in spike, could you please help what can be the reason for it and how to prevent it.

Also, for indices we are going with the default sharding mechanism(5 primary shards) and I believe this might be the reason for it.

Below is the AWS cluster health dashboard of our domain:

Thanks,
Mehak

[warkolm (Mark Walkom)]

Is this the AWS service, or running on your own EC2 instances?
What version>?

[mehak]

Apologies for not mentioning earlier. This is AWS ElasticSearch service and version is 7.4.

[warkolm (Mark Walkom)]

Ok, that's pretty limited with what we can help with due to the changes they have made to it, and the monitoring they offer. You may also want to ask on the AWS forums.

Can you drill down to see what specific index/indices have these spikes?

[mehak]

Okay, sure. Will do that as well.

Yes, this is happening for one of the indices, even if I run a normal search query on my index, it showed a great spike, I am assuming that we haven't really distributed the shards and are just going by the default ones that can be the reason.

[dadoonet (David Pilato)]

Did you look at Cloud by Elastic, also available if needed from AWS Marketplace ?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, Maps UI, Alerting and built-in solutions named Observability, Security, Enterprise Search and what is coming next ...

[agutierrezrodriguez (Antonio Gutiérrez)]

Take a look into JVM heap sizes in the same hour. Seems a garbage collector operation.