I am seeing orders of magnitude difference between the same query run against each individual node. Below are the average query times (where node-5 and node-2 are far more performant) and the allocation output. The datasets are static and there is no indexing into the nodes. Any suggestions on how to troubleshoot this?
ES Version 5.5.1
Server: Red Hat, Amazon Linux AMI release 2017.03
Ave Query time for node-4 6784.0
Ave Query time for node-5 810.6666666666666
Ave Query time for node-2 811.3333333333334
Ave Query time for node-1 6956.333333333333
Ave Query time for node-3 11008.0
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
7 117.7gb 123.2gb 368.7gb 492gb 25 ip ip node-3
6 115.1gb 116.5gb 375.4gb 492gb 23 ip ip node-1
6 115.1gb 116.5gb 375.5gb 492gb 23 ip ip node-5
6 229.6gb 231gb 261gb 492gb 46 ip ip node-4
6 117.3gb 122.9gb 369gb 492gb 24 ip ip node-2