So i'm making a benchmark for performance tests. I have a cluster with 4 machines -> all virtual, all with 64GB RAM and 1.5TB Memory. Only one of those 4 machines is master node.
I Indexed 2 bilion records (each record is 1kb) with 20 shards and replica (so i have now 40 shards)
each shard is 31.1gb size.
The records I indexed has many fields, two of them is id and name.
Now i'm trying to aggragate on id, and sub-aggregate on the name field
this query is 30 seconds long, and when i'm in kibana, on the nodes screen, I see that 2 nodes are with 0% CPU usage, and the other nodes are 10-15%.
It looks strange to me that the query is taking so much time, and the machines doesn't work so hard.
I guess i could add a 5th machine, and i'll get better performance, but i think that with the current hardware I can get also a better search results (a single term query is 10 seconds length)
Am I doing something wrong? Or is it an Elastic limitation?