We Setup ELK stack it's running fine 2 days before. But now We facing that logs are coming delayed . We check logstash log for identify issue. There we getting that
Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover
after some time
Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5
I think the logs you posted are from Logstash, not Filebeat. This typically happens when components downstream (outputs in LS) cannot handle the load you are sending.
Yes you are correct . Can you tell me how we solve this issue ? We need to handle huge log daily basis . So, we need solution which can handle such huge log .
This can be a complex issue because there are a lot of variables in play. From a planning perspective I would identify what the maximum indexing rate is that you want the system to be able to handle without delay. Then in a dev environment independently test each component in the system to make sure that it can handle that load. For example if you are using Elasticsearch then test to make sure it can handle X events per second. Do the same for Logstash (use the stdout output with dots codec and measure events per second with bin/logstash --quiet -f myconfig.conf | pv -Wbart > /dev/null).
If you find that a component cannot handle the load (and you cannot tolerate a processing delay) then scale up horizontally. Add more Logstash instances or more Elasticsearch instances as required.
With your current setup, you can check to see if any of the Logstash worker threads are maxing out the CPU. If this is the case then it could be an issue with your Logstash config, in which case you would want to get a stack trace of Logstash to see what's going on.
If it's not an issue with Logstash, then I would look at your outputs. If you are using Elasticsearch, then use Marvel to look at the percentage of documents being rejected. It is perfectly fine to have some documents rejected but anything higher then 10-15% on a regular basis is a good indication the cluster is overloaded.
Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover
Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn
Note : Logstash instance was working fine for moderate load but when load increased drastically we are facing this issue and logstash is unable to come out of it. To avoid this
we have increased logstash congestion_threshold to 25 also increased -w to 10 for 4 core machine for logstash service still it is in same state.
@santhu227 please create another topic to not mix contexts. Btw. your problem might not be related to logstash per se, but some output (e.g. database or elasticsearch) in logstash not processing events in time.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.