I have 12 indices in one ES cluster (3 dedicated master nodes + 10 C4.4Xlarge data nodes). Each index has about 3.5 billions data and size is about 180GB. One index is allocated 8 primary shards + 1 replica. The node C4.4Xlarge has 32GB memory and 16 vcores, and I'm using AWS Provisioned EBS.
When I try to query one index with terms aggs, it takes 20 seconds to respond, and the ReadThroughput is very high. If I scale out the data nodes to 20, it only takes 1 seconds.
Is it normal under such configuration?
Does this mean that I need more memory so that more data can be cached in disk-cache? Or should I add more primary shards? Or ...
Any suggestions? Thanks!