Logstash - High Network Bandwidth output using S3 Output

Elastic Stack Logstash
1

Hi,

Thanks in advance.

We have two aws logstash instances sending beats traffic to two s3 aws buckets using the s3 output plugin, however are seeing high outbound network traffic.

Id expect to see 1:2 inbound to outbound ratio as we are sending the data to two aws s3 buckets however we are seeing around 1:5 ratio. Is this normal behaviour for the s3 output plugin or are there any other options we can add to limit this output?

Logstash version - v8.12

Example of our logstash config -

input {
beats {
}
}

output {
s3 {
codec => "json_lines"
region => region
bucket => s3_archive_bucket
rotation_strategy => "size_and_time"
size_file => 10485760
time_file => 5
temporary_directory => "/tmp/logstash"
}

s3 {
codec => "json_lines"
region => region
bucket => s3_bucket
rotation_strategy => "size_and_time"
size_file => 10485760
time_file => 5
temporary_directory => "/tmp/logstash"
}
}

2

How did you measured this? This ratio is related to bytes or events? It is not clear.

Please share more context and some evidence.

3

We are seeing this via our infrastructure monitoring tools and AWS portal monitoring.
Logstash Events Handled in/out are at 1:1 ratio, the instances can have been 1-2k active connections throughout the day.

Example of monitoring metrics at a specific point in time -
network bytes in 110m
network bytes out 2.3gb
network packets in 250k
network packets out 940k

4

Maybe it is related to compression.

The communication between Beats and Logstash is compressed by default, but when Logstash writes to S3, it will not be compressed.

I do not use this output, but from the documentation it seens that you can change the encoding to use gzip, so this could reduce the volume of outbound traffic, but the output would be gzipped.

Here is the documentation.

5

Why is more?
As Leandro said, FB send messages in lines, unparsed+ few FB fields. LS will add few more LS fields: your transformations + raw FB message + event +ecs + s3 fields. You can try to enable gzip, and remove unnecessary fields. You can remove it:

mutate{ remove_field => [ "message", "event", "host", "agent", "input", "ecs"] }

First create index/template with all fields, test with 5-10 messages only, set the ruby debug mode, if you need LS log.level=debug and then unnecessary remove fields in LS. For unknown fields such as event check ECS.

Be aware you can lost some important fields, which will might be useful later: host, message/event, etc.