# Elasticsearch 7 has 4x higher AZ transfer costs in AWS compared to 2.4
**URL:** https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745
**Category:** Elasticsearch
**Created:** [March 30, 2020, 8:08pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745 "2020-03-30T20:08:38Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [March 30, 2020, 8:08pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/1 "2020-03-30T20:08:39Z")
Since upgrading to Elasticsearch 7.4.1 from 2.4. (I know 😛 ) we are seeing our AWS availability zone transfer charges skyrocket. Both the old and new clusters used the `cluster.cloud.allocation.awareness.attributes: aws_availability_zone` setting and are deployed across 3 AZ's within an AWS region.
What I'm wondering is, did we miss some "compress internal communication" setting or similar for shard shuffling?
Does ES7 move more shards than 2.4? We have `cluster.routing.allocation.node_concurrent_recoveries:` set to 2 for both, but didn't notice much difference if we went to 8.
Found this [https://www.theguardian.com/info/2020/feb/04/taming-data-transfer-costs-with-elasticsearch](https://www.theguardian.com/info/2020/feb/04/taming-data-transfer-costs-with-elasticsearch) but they seemed to be having problems with queries costing more.
Our costs are directly linked to the data nodes and appears to be them talking to each other.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 30, 2020, 8:36pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/2 "2020-03-30T20:36:52Z")
Have a look at [the docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/modules-transport.html), especially the `transport.compress` setting.
---
### Author:  [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [April 7, 2020, 8:04pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/3 "2020-04-07T20:04:29Z")
So we set up a little lab:
ES2 - 2 data nodes in different AZs
ES7 - 2 data nodes in different AZs
ES7 with `transport.compress` true - 2 data nodes in different AZs
Ingested some data, then deleted it ready for our tests. Using packetbeats to monitor port 9300 on the data nodes.
ES2 idle before we start re-ingestion transferred 156 MB between nodes
ES7 idle transferred 60GB!!
ES7 idle with compression transferred 25GB!!
Re-checking our numbers now, but that's crazy. What is it sending with no indices that's 60GB?
Going to check again with self-monitoring off on ES7 and some other things that may be different between 2-\>7
---
### Author:  [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [April 8, 2020, 3:38pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/4 "2020-04-08T15:38:33Z")
So this is very telling ... from a blank slate, we ingested the same set of data (Shakespeares works) into both ES2 and ES7 (with compression on)
The initial ingestion is similar but then suddenly the data nodes on ES7 continuously transfer data around ... and ES2 is done.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 8, 2020, 3:52pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/5 "2020-04-08T15:52:13Z")
Yes, this seems unexpected indeed. Can you disable compression and grab a full packet capture of the network traffic in this experiment using `tcpdump`? I'm very curious why we're transferring MBs of data so frequently.
---
### Author:  [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [April 8, 2020, 4:13pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/6 "2020-04-08T16:13:18Z")
For sure. What's the best way to get that to you?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 8, 2020, 4:27pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/7 "2020-04-08T16:27:40Z")
> [@Doc\_Kaos](#):
>
> For sure. What's the best way to get that to you?
I sent you a private message with an upload link.
---
### Author:  [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [April 8, 2020, 5:57pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/8 "2020-04-08T17:57:26Z")
Used the following on the 2 data nodes:
```auto
# timeout 30 tcpdump -vv -i any -w data2.cap port 9300
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
1779 packets captured
1788 packets received by filter
0 packets dropped by kernel
```
Hope that works for you?
---
### Author:  [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [April 8, 2020, 6:05pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/9 "2020-04-08T18:05:04Z")
I'm seeing quite a few of these: [https://github.com/elastic/elasticsearch/pull/38262](https://github.com/elastic/elasticsearch/pull/38262)
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 8, 2020, 7:25pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/10 "2020-04-08T19:25:08Z")
I don't see any retention lease sync actions. The only actions I see are these, along with their counts:
```nohighlight
17 cluster:monitor/nodes/info[n]
6 cluster:monitor/nodes/stats[n]
3 cluster:monitor/stats[n]
14 indices:data/read/get[s]
4 indices:data/write/bulk[s][r]
3 indices:monitor/recovery[n]
7 indices:monitor/stats[n]
```
There look to be several hundred indices in this cluster so the index stats and recovery stats are ~200kB, so that's more than half of the 4MB file you shared since something's requesting those stats periodically. There's some write traffic to `monitoring-beats-7-2020.04.08` and some read traffic from Kibana. I don't see anything particularly unexpected here, however, Elasticsearch is apparently doing quite a bit of work to serve client requests.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 8, 2020, 7:31pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/11 "2020-04-08T19:31:54Z")
Ah, sorry, I had a bug in my filter. I do now see retention lease syncs:
```nohighlight
17 cluster:monitor/nodes/info[n]
6 cluster:monitor/nodes/stats[n]
3 cluster:monitor/stats[n]
3 cluster:monitor/xpack/analytics/stats[n]
3 cluster:monitor/xpack/sql/stats/dist[n]
14 indices:data/read/get[s]
4 indices:data/write/bulk[s][r]
3 indices:monitor/recovery[n]
7 indices:monitor/stats[n]
264 indices:admin/seq_no/retention_lease_background_sync[r]
4 indices:admin/seq_no/global_checkpoint_sync[r]
18 indices:data/read/search[phase/query]
```
264 is not so many given how many indices there are, and each one is only a few hundred bytes, so I don't think that's particularly significant.
---
### Author:  [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [April 9, 2020, 2:28pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/12 "2020-04-09T14:28:46Z")
Based on your results there (Is that tool available somewhere? 😄 ) we set our `xpack.monitoring.interval` to 60s from the default (10s). I think you can see where that happened! That should help for sure. We may even go 120s ...
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [April 9, 2020, 3:36pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/13 "2020-04-09T15:36:40Z")
> [@Doc\_Kaos](#):
>
> Is that tool available somewhere? 😄
You probably have it already. I just used `tcpflow` to split the pcap file up, then `cat * | strings | grep '\(cluster\|indices\):' | sort | uniq -c` to find all the things that looked like action names. Then a few spot checks with Wireshark (Statistics -\> TCP flow graphs) to measure some approximate message sizes.
I'm not sure there's too much value in optimising monitoring traffic as you suggest: a few MB per minute is pennies per day in cross-AZ traffic costs, and normally completely swamped by actual production traffic. I don't think this is the reason for the 60GB of traffic that you mentioned above.
---
### Author:  [@Doc\_Kaos](https://discuss.elastic.co/u/Doc_Kaos)
#### Post date: [May 5, 2020, 4:21pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/14 "2020-05-05T16:21:16Z")
Last update, with `transport.compress` on we are seeing the cost benefits we were hoping for with only a slight increase in CPU. Still not sure what was happening with the lab tests, but in production we seem to be at an acceptable data rate.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 2, 2020, 4:21pm UTC](https://discuss.elastic.co/t/elasticsearch-7-has-4x-higher-az-transfer-costs-in-aws-compared-to-2-4/225745/15 "2020-06-02T16:21:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.