# How to handle errors with logstash?

**URL:** <https://discuss.elastic.co/t/how-to-handle-errors-with-logstash/44174>\
**Category:** Logstash\
**Created:** [March 11, 2016, 3:40pm UTC](https://discuss.elastic.co/t/how-to-handle-errors-with-logstash/44174 "2016-03-11T15:40:59Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mehmet\_Ozer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehmet_ozer/32/81567_2.png) [@Mehmet\_Ozer](https://discuss.elastic.co/u/Mehmet_Ozer)\
**Post date:** [March 11, 2016, 3:40pm UTC](https://discuss.elastic.co/t/how-to-handle-errors-with-logstash/44174/1 "2016-03-11T15:40:59Z")

</div>

Hello everyone,

I just a have a question about error handling in Logstash.

For example I have a cluster like;

kafka -\> logstash -\> Elasticsearch

and this logstash is also connected to a HDFS.

so either

**kafka -\> logstash -\> Elasticsearch**  
or  
**kafka -\> logstash -\> HDFS**  
(by the way i am consuming from two different topics)

**my question is here; how can i stop consuming from the topic of HDFS in Kafka if HDFS is not reachable and same for topic of elasticsearch?**

**I am sure that if I use Spark Streaming instead of logstash there is a system of error handling** which will enable me to do so but I dont know at all how it can be done via Logstash. Can you please give me some information about that ?

Thank you all

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 15, 2016, 8:23pm UTC](https://discuss.elastic.co/t/how-to-handle-errors-with-logstash/44174/2 "2016-03-15T20:23:09Z")

</div>

> my question is here; how can i stop consuming from the topic of HDFS in Kafka if HDFS is not reachable and same for topic of elasticsearch?

That's what Logstash does. If one or more outputs are blocked the whole Logstash pipeline stalls and stops reading from the inputs.

---

<div class="post-metadata">

**Author:** ![Joe\_Lawson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joe_lawson/32/3390_2.png) [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)\
**Post date:** [March 24, 2016, 5:59pm UTC](https://discuss.elastic.co/t/how-to-handle-errors-with-logstash/44174/3 "2016-03-24T17:59:23Z")

</div>

Check out the pipeline:  
[https://www.elastic.co/guide/en/logstash/current/pipeline.html](https://www.elastic.co/guide/en/logstash/current/pipeline.html)

Events are created by input threads and passed through a zero length queue  
to a worker that performs filter/output so you have two input threads and  
ES and HDFS output worker threads. Logstash relies on "backpressure" from  
outputs to handle problems where an output isn't working. If one of the  
systems isn't responding you eventually will have blocked output threads  
from that system and the pipeline would build back pressure (ceases  
processing) until the blocking stops.

So how do you stop consumption for either? It just happens if the system is  
unavailable. Keep in mind, if you had multiple outputs with multiple  
streams, if one system goes down on the outputs, it will likely stop the  
processing of all of your events.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:05am UTC](https://discuss.elastic.co/t/how-to-handle-errors-with-logstash/44174/4 "2017-07-06T05:05:29Z")

</div>