# Reg: Issues Monitoring logstash - XPack **URL:** \ **Category:** Logstash\ **Created:** [February 24, 2018, 1:39am UTC](https://discuss.elastic.co/t/reg-issues-monitoring-logstash-xpack/121309 "2018-02-24T01:39:24Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [February 24, 2018, 1:39am UTC](https://discuss.elastic.co/t/reg-issues-monitoring-logstash-xpack/121309/1 "2018-02-24T01:39:25Z")
Trying to alert when Logstash fails on any pipeline or sends data to dead letter queue. Queries: 1. How to find a Pipeline is not running from XPack/monitoring when it is not logging any detail if pipeline is wrongly configured for inputs/output 2. How to find a Pipeline is sending data to dead-letter-queue \<\\> Environment Background: Logstash 6.1.1 + Xpack ES 6.1.1+ Xpack Kibana 6.1.1+ Xpack Logstash Pipeline setup (only pipeline): FileBeat -- {Redis -- grok -- date - ESoutput} - default .monitoring pipline that logstash internally start for sending monitoring data **Scenario and Observations** _Scenario: Force fail ESoutput in pipeline by providing wrong DNS/IP, start Logstash._ The only type of data that came to .monitoring-logstash-6... indice is "logstash\_state", which did not give any pipeline status/vertice status. Regardless, no error or failure info is captured here Below error is observed in logstash log. [2018-02-24T00:24:56,304][FATAL][logstash.runner] An unexpected error occurred! {:error=\>#\, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:213:in`get\_event\_type'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:165:in `event_action_params'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:39:in`event\_action\_tuple'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `block in multi_receive'", "org/jruby/RubyArray.java:2486:in`map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/common.rb:34:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in`multi\_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output\_delegator.rb:50:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:487:in`block in output\_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:486:in`output\_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:438:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:393:in`block in start\_workers'"]} _Scenario: Force fail both ESOutput and Redis input viz. Made redis not reachable by giving wrong key_ THe error as referenced above stopped, and below was observed in logstash log [2018-02-24T01:21:36,301][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin. Pipeline\_id:xxxxxxxxxxxxxxxxxxx Plugin: \"xxxxxxxxxxxxx", password=\>, port=\>6381, data\_type=\>"list", key=\>"xxxxxxxxxxxx", threads=\>8, id=\>"f7c67d2e217cfb37fc070f3aa51df4d2abfe0ab1fa82d566b7cf1d2220d21b0d", enable\_metric=\>true, codec=\>\"json\_9d171ba3-499f-42fe-a184-287e003767c0", enable\_metric=\>true, charset=\>"UTF-8"\>, db=\>0, timeout=\>5, batch\_count=\>125\> Error: closed stream Exception: IOError Stack: org/jruby/RubyIO.java:1364:in `write_nonblock' ...... /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-redis-3.1.6/lib/logstash/inputs/redis.rb:189:in`list\_batch\_listener' org/jruby/RubyMethod.java:119:in `call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-redis-3.1.6/lib/logstash/inputs/redis.rb:175:in`list\_runner' org/jruby/RubyMethod.java:115:in `call' /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-redis-3.1.6/lib/logstash/inputs/redis.rb:94:in`run' /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:524:in `inputworker' /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:517:in`block in start\_input' Regardless - The issue are - No information regarding failure is captured as part of monitoring indices - The failures value is "0" for any logstash\_stats that are captured i indice, but it is for reload though - CURL api giving error as below so cannot use to check what happened to pipeline curl -XGET 'localhost:9600/\_node/stats/pipelines' {"status":500,"request\_method":"GET","path\_info":"/\_node/stats/pipelines","query\_string":"","http\_version":"HTTP/1.1","http\_accept":"_/_","error":"Unexpected Internal Error","class":"LogStash::Instrument::MetricStore::MetricNotFound","message":"For path: events. Map keys: [:pipelines, :reloads]","backtrace":["/usr/share/logstash/logstash-core/lib/logstash/instrument/metric\_store.rb:225:in `block in get_recursively'","org/jruby/RubyArray.java:1734:in`each'","/usr/share/logstash/logstash-core/lib/logstash/instrument/metric\_store.rb:224:in `get_recursively'","/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:235:in`block in get\_recursively'","org/jruby/RubyArray.java:1734:in `each'","/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:224:in`get\_recursively'","/usr/share/logstash/logstash-core/lib/logstash/instrument/metric\_store.rb:95:in `block in get'","org/jruby/ext/thread/Mutex.java:148:in`synchronize'","/usr/share/logstash/logstash-core/lib/logstash/instrument/metric\_store.rb:94:in `get'","/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:108:in`get\_shallow'","/usr/share/logstash/logstash-core/lib/log sometimes I get below as well, with logstash automatically restarting - and starts .monitoring... internal pipeline to send monitoring data. curl -XGET 'localhost:9600/\_node/stats/pipelines' curl: (7) Failed connect to localhost:9600; Connection refused Thanks for guidance. ---
**Author:** ![Karthik\_Ramachandran](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@Karthik\_Ramachandran](https://discuss.elastic.co/u/Karthik_Ramachandran)\ **Post date:** [February 26, 2018, 6:38pm UTC](https://discuss.elastic.co/t/reg-issues-monitoring-logstash-xpack/121309/2 "2018-02-26T18:38:50Z")
Rechecking as I didn't any response. It would be helpful if someone can guide me on how to find logstash pipeline failure. For now, I'm planning to use the target Es Indice to monitor last arrived data to alert on failure, but it will work only if the data to indice is coming from only one instance. ---
**Author:** ![Bill\_McConaghy](https://avatars.discourse-cdn.com/v4/letter/b/ed655f/32.png) [@Bill\_McConaghy](https://discuss.elastic.co/u/Bill_McConaghy)\ **Post date:** [March 15, 2018, 8:12pm UTC](https://discuss.elastic.co/t/reg-issues-monitoring-logstash-xpack/121309/3 "2018-03-15T20:12:44Z")
I would suggest asking this question in the Logstash forum. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 12, 2018, 8:13pm UTC](https://discuss.elastic.co/t/reg-issues-monitoring-logstash-xpack/121309/4 "2018-04-12T20:13:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.