# Logstash - High Network Bandwidth output using S3 Output

**URL:** <https://discuss.elastic.co/t/logstash-high-network-bandwidth-output-using-s3-output/366422>\
**Category:** Logstash\
**Created:** [September 11, 2024, 4:11pm UTC](https://discuss.elastic.co/t/logstash-high-network-bandwidth-output-using-s3-output/366422 "2024-09-11T16:11:52Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![iseeldur](https://avatars.discourse-cdn.com/v4/letter/i/dc4da7/32.png) [@iseeldur](https://discuss.elastic.co/u/iseeldur)\
**Post date:** [September 11, 2024, 4:11pm UTC](https://discuss.elastic.co/t/logstash-high-network-bandwidth-output-using-s3-output/366422/1 "2024-09-11T16:11:52Z")

</div>

Hi,

Thanks in advance.

We have two aws logstash instances sending beats traffic to two s3 aws buckets using the s3 output plugin, however are seeing high outbound network traffic.

Id expect to see 1:2 inbound to outbound ratio as we are sending the data to two aws s3 buckets however we are seeing around 1:5 ratio. Is this normal behaviour for the s3 output plugin or are there any other options we can add to limit this output?

Logstash version - v8.12

Example of our logstash config -

input {  
beats {  
}  
}

output {  
s3 {  
codec =\> "json\_lines"  
region =\> region  
bucket =\> s3\_archive\_bucket  
rotation\_strategy =\> "size\_and\_time"  
size\_file =\> 10485760  
time\_file =\> 5  
temporary\_directory =\> "/tmp/logstash"  
}

s3 {  
codec =\> "json\_lines"  
region =\> region  
bucket =\> s3\_bucket  
rotation\_strategy =\> "size\_and\_time"  
size\_file =\> 10485760  
time\_file =\> 5  
temporary\_directory =\> "/tmp/logstash"  
}  
}

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 11, 2024, 4:37pm UTC](https://discuss.elastic.co/t/logstash-high-network-bandwidth-output-using-s3-output/366422/2 "2024-09-11T16:37:16Z")

</div>

> [@iseeldur](#):
>
> Id expect to see 1:2 inbound to outbound ratio as we are sending the data to two aws s3 buckets however we are seeing around 1:5 ratio

How did you measured this? This ratio is related to bytes or events? It is not clear.

Please share more context and some evidence.

---

<div class="post-metadata">

**Author:** ![iseeldur](https://avatars.discourse-cdn.com/v4/letter/i/dc4da7/32.png) [@iseeldur](https://discuss.elastic.co/u/iseeldur)\
**Post date:** [September 11, 2024, 7:06pm UTC](https://discuss.elastic.co/t/logstash-high-network-bandwidth-output-using-s3-output/366422/3 "2024-09-11T19:06:20Z")

</div>

We are seeing this via our infrastructure monitoring tools and AWS portal monitoring.  
Logstash Events Handled in/out are at 1:1 ratio, the instances can have been 1-2k active connections throughout the day.

Example of monitoring metrics at a specific point in time -  
network bytes in 110m  
network bytes out 2.3gb  
network packets in 250k  
network packets out 940k

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 11, 2024, 7:17pm UTC](https://discuss.elastic.co/t/logstash-high-network-bandwidth-output-using-s3-output/366422/4 "2024-09-11T19:17:37Z")

</div>

Maybe it is related to compression.

The communication between Beats and Logstash is compressed by default, but when Logstash writes to S3, it will not be compressed.

I do not use this output, but from the documentation it seens that you can change the `encoding` to use `gzip`, so this could reduce the volume of outbound traffic, but the output would be gzipped.

Here is the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html#plugins-outputs-s3-encoding).

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [September 12, 2024, 6:29am UTC](https://discuss.elastic.co/t/logstash-high-network-bandwidth-output-using-s3-output/366422/5 "2024-09-12T06:29:39Z")

</div>

Why is more?  
As Leandro said, FB send messages in lines, unparsed+ few FB fields. LS will add few more LS fields: your transformations + raw FB message + event +ecs + s3 fields. You can try to enable gzip, and remove unnecessary fields. You can remove it:

` mutate{ remove_field => ["message", "event", "host", "agent", "input", "ecs"] }`

First create index/template with all fields, test with 5-10 messages only, set the ruby debug mode, if you need LS `log.level=debug` and then unnecessary remove fields in LS. For unknown fields such as event check [ECS](https://www.elastic.co/guide/en/ecs/current/ecs-event.html).

Be aware you can lost some important fields, which will might be useful later: host, message/event, etc.