We have an Elasticsearch 7.1 cluster (AWS) with 2 indexes. The parent index has one shard and two replicas, whereas the child index has 3 shards and 2 replicas. Right now we’re running on 8 data nodes and the shard allocation is as follows:
node 1 Parent 0 (R) Child 2 (P) node 2 Parent 0 (P) Child 1 (P) node 3 Parent 0 (R) Child 2 (R) node 4 Child 0 (P) Child 2 (R) node 5 Child 1 (R) Kibana (R) node 6 Child 1 (R) Kibana (P) node 7 Child 0 (R) node 8 Child 0 (R)
We’re seeing that some of the nodes (4-8) aren’t participating in search requests. Is this because we have too many nodes? Will this hurt performance?
We also really don't have enough data (35GB) to justify 3 shards on the child index but left room for growth. Could this also hurt performance?
The cluster performs ok most of the time, but when there's a spike in traffic clients experience 504 gateway timeouts for a few minutes. I'm trying to determine if the cluster configuration contributes to this or if its a load balancer issue.
Any help is greatly appreciated!