Logstash consuming too much bandwidth

[Jitendra_Kumhar]

Hello,
I am sending logs from filebeat to Logstash and then from logstash to ElasticSearch.
The problem is Logstash is consuming too much bandwidth.I have attached network usage graph of logstash server.
(green is incoming from filebeat)

In the starting I am just writing to file on Logstash server instead of sending to ElasticSearch.You can see the incoming rate.Then when I turn on ElasticSearch output.The incoming rate decreases and also outgoing is 10 times more data.
Can anyone help please.What configurations changes or hardware improvements should I make?

[Christian_Dahlqvist]

You may want to enable http_compression for the Elasticsearch output if you have not already.

[Jitendra_Kumhar]

Thank You for the reply
But why is my incoming data rate reduced when elasticsearch output is enabled?

[Christian_Dahlqvist]

Maybe Elasticsearch is not able to keep up, which results in back pressure being applied?

[Jitendra_Kumhar]

Currently I have 1 ingest node,1 master node and 2 data nodes.
I added persistent queues in logstash of 30 gb.They gets filled up quickly.
So the reason is mostly back pressure by ElasticSearch.
Do you have any suggestion how do I handle this?

[Christian_Dahlqvist]

I would recommend adding resources to the Elasticsearch cluster. What is the current hardware specification?

[Jitendra_Kumhar]

Ingest node:

Model name: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
Stepping: 3
CPU MHz: 3000.000
BogoMIPS: 6000.00
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 25344K

Ram : 3.549 GB

Total disk : 31 GB

Data Node(2 x):

Ram : 31.548 GB
Total disk : 2.3 TB
Master Node

Ram : 3.549 GB

Total disk : 31 GB

[Jitendra_Kumhar]

Any suggestion?

[rugenl (Len Rugen)]

Look at the logs @18:04, when the lines crossed. How many events per second were ingesting before that time?

I think you need to determine if you have an ingest error condition before assuming it's a capacity issue.

[Jitendra_Kumhar]

How do I check how many events ingested at that time ?

[rugenl (Len Rugen)]

Kibana monitoring, if you had it enabled.

[Jitendra_Kumhar]

Hey,
I enabled http_compression on logstash.
Now that machines network monitoring looks like this

The graph is MiB per minute
Green is receiving and yellow is transmitting.I don't understand why it is exactly asymmetric.I have done speed test on this machine and it can give speed upto 500 Mb/s download and 500 Mb/s upload speed.