We have an 18 Node cluster, usually one shard per node exists. This is deployed in AWS with r3.xlarge instances. we have two cordinating nodes to add to this
we recently in one of our test increased our search rate from ~240 sec to ~360 sec and see that our response times from ES more than doubled. With the existing config I expected the cluster to perform the same with the increase of load as none of the resource such as memory (jvm and node) and cpu seems maxed, there was minimal or no indexing during this time. I puzzled as to whats caused the increase in time. Any help is greatly appreciated
Shows how the increased request rate affected time , the far right has the normal load and left has increased load
stats api output
form the API
all query, fetch and get seems to take about 10 ms is this normal
transsport for the nodes reads like bleow is this normal?
rx_count: 1102435,
rx_size_in_bytes: 77596791987,
tx_count: 1102436,
tx_size_in_bytes: 2197916623
What other reason could cause the spike ?