# Elasticsearch diminishes throughput behind a load balancer.... why? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 21, 2013, 4:57pm UTC](https://discuss.elastic.co/t/elasticsearch-diminishes-throughput-behind-a-load-balancer-why/14520 "2013-11-21T16:57:05Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Dario\_Rossi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dario_rossi/32/1672_2.png) [@Dario\_Rossi](https://discuss.elastic.co/u/Dario_Rossi)\ **Post date:** [November 21, 2013, 4:57pm UTC](https://discuss.elastic.co/t/elasticsearch-diminishes-throughput-behind-a-load-balancer-why/14520/1 "2013-11-21T16:57:05Z")
So I have two instances of ES in production and we put them in front of a load balancer that does round robin balancing and access control. I know that the recommended way to balance ES is to use a non data node, but we need access control. So these are the results after some testing with JMeter: WITHOUT LOAD BALANCER (only one instance): Throughput 165M requests/minute, ~28ms average latency. WITH THE LOAD BALANCER (one here would expect the throughput to roughly double and latency to increase) TP 124M req/minute and ~85ms average latency.... And that's unexepected... because I am allright with the latency increasing, but I am not allright with the throughput decreasing. We are using NGINX to load balance and access control and I've set index.auto\_expand\_replicas to 0-all Any ideas why I am getting this result? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 21, 2013, 5:10pm UTC](https://discuss.elastic.co/t/elasticsearch-diminishes-throughput-behind-a-load-balancer-why/14520/2 "2013-11-21T17:10:05Z")
Is the time you measure come from `took` field? Or is it the time you can see on the client side? Could you measure both? Does Nginx use persistent connections? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 21 novembre 2013 at 17:57:11, Dario Rossi ([darioros@gmail.com](mailto:darioros@gmail.com)) a écrit: So I have two instances of ES in production and we put them in front of a load balancer that does round robin balancing and access control. I know that the recommended way to balance ES is to use a non data node, but we need access control. So these are the results after some testing with JMeter: WITHOUT LOAD BALANCER (only one instance): Throughput 165M requests/minute, ~28ms average latency. WITH THE LOAD BALANCER (one here would expect the throughput to roughly double and latency to increase) TP 124M req/minute and ~85ms average latency.... And that's unexepected... because I am allright with the latency increasing, but I am not allright with the throughput decreasing. We are using NGINX to load balance and access control and I've set index.auto\_expand\_replicas to 0-all ## Any ideas why I am getting this result? You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Dario\_Rossi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dario_rossi/32/1672_2.png) [@Dario\_Rossi](https://discuss.elastic.co/u/Dario_Rossi)\ **Post date:** [November 21, 2013, 5:32pm UTC](https://discuss.elastic.co/t/elasticsearch-diminishes-throughput-behind-a-load-balancer-why/14520/3 "2013-11-21T17:32:50Z")
The time measure comes from JMeter and it is the average response time (from connection open to close). So yes, \*this is the time on client side. \* I'm not interested in the "took" field. I don't know if NGINX uses persistent connections... I've to ask the techops. Will do tomorrow (he's gone ...). Why? What would be the best arrangement? Il giorno giovedì 21 novembre 2013 17:10:05 UTC, David Pilato ha scritto: > Is the time you measure come from `took` field? > Or is it the time you can see on the client side? > > Could you measure both? > > Does Nginx use persistent connections? > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet[https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Fdadoonet&sa=D&sntz=1&usg=AFQjCNE-DMC3YEu3X\_lhRIhUzuSZGsaSqA](https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Fdadoonet&sa=D&sntz=1&usg=AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA) > | @elasticsearchfr[https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Felasticsearchfr&sa=D&sntz=1&usg=AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA](https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Felasticsearchfr&sa=D&sntz=1&usg=AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA) > > Le 21 novembre 2013 at 17:57:11, Dario Rossi ([dari...@gmail.com](mailto:dari...@gmail.com)\) > a écrit: > > So I have two instances of ES in production and we put them in front of a > load balancer that does round robin balancing and access control. I know > that the recommended way to balance ES is to use a non data node, but we > need access control. > > So these are the results after some testing with JMeter: > > WITHOUT LOAD BALANCER (only one instance): > > Throughput 165M requests/minute, ~28ms average latency. > > WITH THE LOAD BALANCER (one here would expect the throughput to roughly > double and latency to increase) > > TP 124M req/minute and ~85ms average latency.... > > And that's unexepected... because I am allright with the latency > increasing, but I am not allright with the throughput decreasing. > > We are using NGINX to load balance and access control and I've set index.auto\_expand\_replicas > to 0-all > > ## Any ideas why I am getting this result? > > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 21, 2013, 5:35pm UTC](https://discuss.elastic.co/t/elasticsearch-diminishes-throughput-behind-a-load-balancer-why/14520/4 "2013-11-21T17:35:22Z")
Because if you are opening a new connection each time you send a request you probably are spending some time on that. That said, I am interested in the mean value of took field. I mean that it could help to build the right diagnostic: is the problem coming from elasticsearch or from NGinx? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 21 novembre 2013 at 18:32:54, Dario Rossi ([darioros@gmail.com](mailto:darioros@gmail.com)) a écrit: The time measure comes from JMeter and it is the average response time (from connection open to close). So yes, this is the time on client side. I'm not interested in the "took" field. I don't know if NGINX uses persistent connections... I've to ask the techops. Will do tomorrow (he's gone ...). Why? What would be the best arrangement? Il giorno giovedì 21 novembre 2013 17:10:05 UTC, David Pilato ha scritto: Is the time you measure come from `took` field? Or is it the time you can see on the client side? Could you measure both? Does Nginx use persistent connections? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr Le 21 novembre 2013 at 17:57:11, Dario Rossi ([dari...@gmail.com](mailto:dari...@gmail.com)) a écrit: So I have two instances of ES in production and we put them in front of a load balancer that does round robin balancing and access control. I know that the recommended way to balance ES is to use a non data node, but we need access control. So these are the results after some testing with JMeter: WITHOUT LOAD BALANCER (only one instance): Throughput 165M requests/minute, ~28ms average latency. WITH THE LOAD BALANCER (one here would expect the throughput to roughly double and latency to increase) TP 124M req/minute and ~85ms average latency.... And that's unexepected... because I am allright with the latency increasing, but I am not allright with the throughput decreasing. We are using NGINX to load balance and access control and I've set index.auto\_expand\_replicas to 0-all ## Any ideas why I am getting this result? ## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Dario\_Rossi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dario_rossi/32/1672_2.png) [@Dario\_Rossi](https://discuss.elastic.co/u/Dario_Rossi)\ **Post date:** [November 22, 2013, 9:39am UTC](https://discuss.elastic.co/t/elasticsearch-diminishes-throughput-behind-a-load-balancer-why/14520/5 "2013-11-22T09:39:38Z")
I really think the problem comes from NGINX because if I do the test by sending request to a machine of the cluster and bypassing NGINX it is much faster. Il giorno giovedì 21 novembre 2013 17:35:22 UTC, David Pilato ha scritto: > Because if you are opening a new connection each time you send a request > you probably are spending some time on that. > > That said, I am interested in the mean value of took field. > I mean that it could help to build the right diagnostic: is the problem > coming from elasticsearch or from NGinx? > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet[https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Fdadoonet&sa=D&sntz=1&usg=AFQjCNE-DMC3YEu3X\_lhRIhUzuSZGsaSqA](https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Fdadoonet&sa=D&sntz=1&usg=AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA) > | @elasticsearchfr[https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Felasticsearchfr&sa=D&sntz=1&usg=AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA](https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Felasticsearchfr&sa=D&sntz=1&usg=AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA) > > Le 21 novembre 2013 at 18:32:54, Dario Rossi ([dari...@gmail.com](mailto:dari...@gmail.com)\) > a écrit: > > The time measure comes from JMeter and it is the average response time > (from connection open to close). So yes, \*this is the time on client > side. \* > I'm not interested in the "took" field. > > I don't know if NGINX uses persistent connections... I've to ask the > techops. Will do tomorrow (he's gone ...). Why? What would be the best > arrangement? > > Il giorno giovedì 21 novembre 2013 17:10:05 UTC, David Pilato ha scritto: > > > Is the time you measure come from `took` field? > > Or is it the time you can see on the client side? > > > > Could you measure both? > > > > Does Nginx use persistent connections? > > > > ``` > > -- > > > > ``` > > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > > @dadoonet[https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Fdadoonet&sa=D&sntz=1&usg=AFQjCNE-DMC3YEu3X\_lhRIhUzuSZGsaSqA](https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Fdadoonet&sa=D&sntz=1&usg=AFQjCNE-DMC3YEu3X_lhRIhUzuSZGsaSqA) > > | @elasticsearchfr[https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Felasticsearchfr&sa=D&sntz=1&usg=AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA](https://www.google.com/url?q=https%3A%2F%2Ftwitter.com%2Felasticsearchfr&sa=D&sntz=1&usg=AFQjCNGfXdQ98RWFMJXdiqpKnZb5GMg0zA) > > > > Le 21 novembre 2013 at 17:57:11, Dario Rossi ([dari...@gmail.com](mailto:dari...@gmail.com)) a écrit: > > > > So I have two instances of ES in production and we put them in front of > > a load balancer that does round robin balancing and access control. I know > > that the recommended way to balance ES is to use a non data node, but we > > need access control. > > > > So these are the results after some testing with JMeter: > > > > WITHOUT LOAD BALANCER (only one instance): > > > > Throughput 165M requests/minute, ~28ms average latency. > > > > WITH THE LOAD BALANCER (one here would expect the throughput to roughly > > double and latency to increase) > > > > TP 124M req/minute and ~85ms average latency.... > > > > And that's unexepected... because I am allright with the latency > > increasing, but I am not allright with the throughput decreasing. > > > > We are using NGINX to load balance and access control and I've set index.auto\_expand\_replicas > > to 0-all > > > > ## Any ideas why I am getting this result? > > > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:05am UTC](https://discuss.elastic.co/t/elasticsearch-diminishes-throughput-behind-a-load-balancer-why/14520/6 "2017-07-06T02:05:23Z")