# Bluk indexing performance over a high latency newtork **URL:** \ **Category:** Elasticsearch\ **Created:** [July 6, 2016, 3:42pm UTC](https://discuss.elastic.co/t/bluk-indexing-performance-over-a-high-latency-newtork/54844 "2016-07-06T15:42:08Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![eugene\_miretsky](https://avatars.discourse-cdn.com/v4/letter/e/9fc348/32.png) [@eugene\_miretsky](https://discuss.elastic.co/u/eugene_miretsky)\ **Post date:** [July 6, 2016, 3:42pm UTC](https://discuss.elastic.co/t/bluk-indexing-performance-over-a-high-latency-newtork/54844/1 "2016-07-06T15:42:09Z")
We have a 3 node ES cluster set up in AWS. Running the default Rally benchmark from another instance in the same AWS VPC (using --benchmark-only) results in ~45,000 docs/s indexing rate. Running the same benchmark from the VM in out own data center over a VPN results in ~2,500 docs/s. We get even worse performance using our own benchmark - 20,000docs/s vs 200 docs/s. The VPN latency is 100ms - that's a lot, but doesn't quite explain the huge performance difference. If we assume that each batch takes 1s to processes, the latency should add only 0.2s to the end-to-end batch processing time. We are using the default config, so http.keep\_alive is true. The network throughput is ~12MB/s (we are not even close using that). Any advice on how to solve/debug this? @danielmitterdorfer (for Rally advice) ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [July 7, 2016, 4:15am UTC](https://discuss.elastic.co/t/bluk-indexing-performance-over-a-high-latency-newtork/54844/2 "2016-07-07T04:15:28Z")
HI @eugene_miretsky, one (logical) bulk request is not just one network packet but transferred in HTTP chunked transfer encoding, which means that there are multiple network packets. So I guess your assumption that your latency of 100ms adds just 200ms is not true. I'd just capture the network packets to see what's going on. You can use [Wireshark](https://www.wireshark.org/) for that. Daniel ---
**Author:** ![eugene\_miretsky](https://avatars.discourse-cdn.com/v4/letter/e/9fc348/32.png) [@eugene\_miretsky](https://discuss.elastic.co/u/eugene_miretsky)\ **Post date:** [July 7, 2016, 7:50pm UTC](https://discuss.elastic.co/t/bluk-indexing-performance-over-a-high-latency-newtork/54844/3 "2016-07-07T19:50:23Z")
Thanks Daniel! I'm not exactly a networking expert, but gave Wireshark my best shot. Attached are screen shots of Wireshark over 1. Good connection (same VPC in AWS) ![](https://us1.discourse-cdn.com/elastic/original/2X/b/bf98d2d34f27e6a9020ca3324c75fe41c071f1ce.png) 2. slow network 100ms latency connection ![](https://us1.discourse-cdn.com/elastic/original/2X/9/94f7a6f5c201b4a8921fb83e471f8ad59d3b57e4.png) As you can see the packet size is 10x smaller, and there is an ACK being sent for every packet (as opposed to a batch of packets). Also duplicate ACK rate is 3%. Any idea what's causing this? From what I understand ES is using HTTP chunking, and it look like in the latter case the chunks are much smaller. Is there a way to tune this? ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [July 8, 2016, 9:31am UTC](https://discuss.elastic.co/t/bluk-indexing-performance-over-a-high-latency-newtork/54844/4 "2016-07-08T09:31:02Z")
Hi Eugene, just as a heads up: I try to look more closely into this but it could take a bit of time until I can spare some cycles. Daniel ---
**Author:** ![eugene\_miretsky](https://avatars.discourse-cdn.com/v4/letter/e/9fc348/32.png) [@eugene\_miretsky](https://discuss.elastic.co/u/eugene_miretsky)\ **Post date:** [July 13, 2016, 1:29pm UTC](https://discuss.elastic.co/t/bluk-indexing-performance-over-a-high-latency-newtork/54844/5 "2016-07-13T13:29:58Z")
@danielmitterdorfer Sure - any help would be appreciated. Any idea how to enable HTTP chunking and compression in Rally? ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [July 14, 2016, 7:01am UTC](https://discuss.elastic.co/t/bluk-indexing-performance-over-a-high-latency-newtork/54844/6 "2016-07-14T07:01:55Z")
Hi Eugene, do you have the original packet dumps around? Would be great if you could share them with me. Compression is not supported out of the box the Python Elasticsearch client (i.e. it's not possible without writing custom code) but I have some code lying around that does that (as I needed it for some [HTTP compression benchmark in Elasticsearch](https://github.com/elastic/elasticsearch/issues/7309)). I see what I can do to integrate that into Rally. Daniel ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [July 14, 2016, 12:19pm UTC](https://discuss.elastic.co/t/bluk-indexing-performance-over-a-high-latency-newtork/54844/7 "2016-07-14T12:19:38Z")
> Any idea how to enable HTTP chunking and compression in Rally? If you use the latest master you can specify arbitrary client options (see [docs](http://esrally.readthedocs.io/en/latest/command_line_reference.html#client-options)). Otherwise, it will be supported starting with Rally 0.4.0. For convenience: `--client-options="compressed:true,timeout:90,request_timeout:90"`. (the latter two options are needed as you override the default and I assume you don't want to change these values). **Edit:** Expect that query latency will increase with compression (at least it does in my experience). Bulk indexing throughput should be roughly identical. I tried to support for disabling chunking as hinted by Honza in [elasticsearch-py/#422](https://github.com/elastic/elasticsearch-py/issues/422) but it did not work well together with the rest of this feature so it stays unsupported for now. But I am always happy to receive PRs. 😉 Daniel ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:35pm UTC](https://discuss.elastic.co/t/bluk-indexing-performance-over-a-high-latency-newtork/54844/8 "2017-07-05T22:35:21Z")