Performance Decrease with Load - Why?

We have an 18 Node cluster, usually one shard per node exists. This is deployed in AWS with r3.xlarge instances. we have two cordinating nodes to add to this

we recently in one of our test increased our search rate from ~240 sec to ~360 sec and see that our response times from ES more than doubled. With the existing config I expected the cluster to perform the same with the increase of load as none of the resource such as memory (jvm and node) and cpu seems maxed, there was minimal or no indexing during this time. I puzzled as to whats caused the increase in time. Any help is greatly appreciated

Shows how the increased request rate affected time , the far right has the normal load and left has increased load

stats api output

form the API
all query, fetch and get seems to take about 10 ms is this normal

transsport for the nodes reads like bleow is this normal?
rx_count: 1102435,
rx_size_in_bytes: 77596791987,
tx_count: 1102436,
tx_size_in_bytes: 2197916623

What other reason could cause the spike ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.