we are running a cluster of elasticsearch with the following specs:
- we run on AWS EC2's (self managed, not their Saas).
- 3 nodes (all do everything... data, master)
- each node is r4.large type (2 CPU, 15GB RAM), 8GB is for heap.
- SSD disks (AWS EBS gp2 type)
- Elasticsearch 5.6.0
- JVM 1.8.0_151
we have a single index with 4 primary shards and no replicas.
the index has around 3,000,000 documents that weigh around 1GB in total (I know its not much at all... this is why i'm surprised at the performance issues).
and when we run some load testing, with about 40 r\s , the CPU of the nodes goes near 100% (not all the nodes, sometimes one or two, and sometimes all. I guess it depends on the queries or searches we perform at any given moment...)
any idea what can be the issue?
what can we check to better troubleshoot?