I am sending data from various filebeats to two logstash machines and these machines send to ElasticSearch.
When I monitored network it shows 12 MiB/minute receiving and 5 MiB/minute(because I am using http_compression maybe) outgoing.
I have persistent queue of 10 GB on logstash machine which gets filled very fast.
I cant figure out why is this happening.
So is this ElasticSearch back pressuring or something with logstash?
Logstash is only able to send data as fast as Elasticsearch can accept it, and it sounds like back-pressure is being applied. What is the specification of your Elasticsearch cluster?
1 Like
4 data nodes with 32 GB Ram and 2 master nodes with 4 GB Ram.
I am sending to all 4 data nodes from logstash
How much CPU and what kind of storage? Local SSDs?
You should also always have at least 3 master eligible nodes in order to get a highly available cluster.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.