# Logstash pipelines not visible in stack monitoring **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-monitoring\ **Created:** [December 24, 2023, 8:09am UTC](https://discuss.elastic.co/t/logstash-pipelines-not-visible-in-stack-monitoring/349894 "2023-12-24T08:09:33Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![SamehSaeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samehsaeed/32/145262_2.png) [@SamehSaeed](https://discuss.elastic.co/u/SamehSaeed)\ **Post date:** [December 24, 2023, 8:09am UTC](https://discuss.elastic.co/t/logstash-pipelines-not-visible-in-stack-monitoring/349894/1 "2023-12-24T08:09:33Z")
I'm unable to monitor pipelines through kibana ==\> ![Stack Monitoring](https://us1.discourse-cdn.com/elastic/original/3X/7/7/77310abf4a0f6006f49498b46d167bf757ad0d12.png) So i tried to pick elasticsearch, then ingest pipelines and this error popped up upon trying to install elasticsearch integration ==\> ![Elasticsearch Integration Install fail](https://us1.discourse-cdn.com/elastic/original/3X/f/0/f06c562a8b94d4b3e1679e29952f4ca5c6f8ffda.png) How can i monitor pipelines? Is it mandatory to install metric beat? Using windows, ELK stack running as a service (through NSSM) logstash.yml : ```auto pipeline.workers: 1 xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.url: "http://localhost:9200" (the rest of the file is commented) ``` ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [December 24, 2023, 9:17am UTC](https://discuss.elastic.co/t/logstash-pipelines-not-visible-in-stack-monitoring/349894/2 "2023-12-24T09:17:39Z")
If you using LS 7.x you could use [collectors](https://www.elastic.co/guide/en/logstash/current/monitoring-internal-collection-legacy.html#configure-internal-collectors-legacy) internal metrics which deprecated. Starting with 8.x, you should use metricbeat and LS module, which is explained [here](https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html). Also is possible to use [APIs for monitoring](https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#pipeline-stats) in raw JSON format. ---
**Author:** ![SamehSaeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samehsaeed/32/145262_2.png) [@SamehSaeed](https://discuss.elastic.co/u/SamehSaeed)\ **Post date:** [December 24, 2023, 11:02am UTC](https://discuss.elastic.co/t/logstash-pipelines-not-visible-in-stack-monitoring/349894/3 "2023-12-24T11:02:40Z")
Thanks Rios, I'll look into it. Regarding monitoring APIs, why does it return all pipelines even if some of them were terminated and not working due to some error? (found through logstash-plain log) Command used through power shell : `Invoke-RestMethod -Method GET -Uri 'http://localhost:9600/_node/pipelines?pretty'` ---
**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\ **Post date:** [December 24, 2023, 2:49pm UTC](https://discuss.elastic.co/t/logstash-pipelines-not-visible-in-stack-monitoring/349894/4 "2023-12-24T14:49:16Z")
From the documentation: > You can see the stats for a specific pipeline by including the pipeline ID. In the following example, the ID of the pipeline is `test`: `curl -XGET 'localhost:9600/_node/stats/pipelines/test?pretty'` ---
**Author:** ![SamehSaeed](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/samehsaeed/32/145262_2.png) [@SamehSaeed](https://discuss.elastic.co/u/SamehSaeed)\ **Post date:** [December 25, 2023, 6:56am UTC](https://discuss.elastic.co/t/logstash-pipelines-not-visible-in-stack-monitoring/349894/5 "2023-12-25T06:56:04Z")
Well I have 72 pipelines, sometimes when running logstash all 72 pipelines would start, while at other times some of them gets terminated (still trying to figure out why) ```auto [2023-12-25T09:41:45,779][ERROR][logstash.javapipeline][bkgetivrnumbersoffers] Pipeline error {:pipeline_id=>"bkgetivrnumbersoffers", :exception=>#, :backtrace=>["org/jruby/RubyModule.java:4309:in `const_missing'", "D:/ELK 8.10.2/logstash-8.10.2/vendor/bundle/jruby/3.1.0/gems/sequel-5.71.0/lib/sequel/adapters/jdbc/oracle.rb:11:in `block in JDBC'", "D:/ELK 8.10.2/logstash-8.10.2/vendor/bundle/jruby/3.1.0/gems/sequel-5.71.0/lib/sequel/adapters/jdbc.rb:379:in `adapter_initialize'", "D:/ELK 8.10.2/logstash-8.10.2/vendor/bundle/jruby/3.1.0/gems/sequel-5.71.0/lib/sequel/database/misc.rb:169:in `initialize'", "org/jruby/RubyClass.java:904:in `new'", "D:/ELK 8.10.2/logstash-8.10.2/vendor/bundle/jruby/3.1.0/gems/sequel-5.71.0/lib/sequel/database/connecting.rb:57:in `connect'", "D:/ELK 8.10.2/logstash-8.10.2/vendor/bundle/jruby/3.1.0/gems/sequel-5.71.0/lib/sequel/core.rb:124:in `connect'", "D:/ELK 8.10.2/logstash-8.10.2/vendor/bundle/jruby/3.1.0/gems/logstash-integration-jdbc-5.4.4/lib/logstash/plugin_mixins/jdbc/jdbc.rb:123:in `block in jdbc_connect'", "org/jruby/RubyKernel.java:1586:in `loop'", "D:/ELK 8.10.2/logstash-8.10.2/vendor/bundle/jruby/3.1.0/gems/logstash-integration-jdbc-5.4.4/lib/logstash/plugin_mixins/jdbc/jdbc.rb:120:in `jdbc_connect'", "D:/ELK 8.10.2/logstash-8.10.2/vendor/bundle/jruby/3.1.0/gems/logstash-integration-jdbc-5.4.4/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in `open_jdbc_connection'", "D:/ELK 8.10.2/logstash-8.10.2/vendor/bundle/jruby/3.1.0/gems/logstash-integration-jdbc-5.4.4/lib/logstash/inputs/jdbc.rb:309:in `register'", "D:/ELK 8.10.2/logstash-8.10.2/vendor/bundle/jruby/3.1.0/gems/logstash-mixin-ecs_compatibility_support-1.3.0-java/lib/logstash/plugin_mixins/ecs_compatibility_support/target_check.rb:48:in `register'", "D:/ELK 8.10.2/logstash-8.10.2/logstash-core/lib/logstash/java_pipeline.rb:237:in `block in register_plugins'", "org/jruby/RubyArray.java:1987:in `each'", "D:/ELK 8.10.2/logstash-8.10.2/logstash-core/lib/logstash/java_pipeline.rb:236:in `register_plugins'", "D:/ELK 8.10.2/logstash-8.10.2/logstash-core/lib/logstash/java_pipeline.rb:395:in `start_inputs'", "D:/ELK 8.10.2/logstash-8.10.2/logstash-core/lib/logstash/java_pipeline.rb:320:in `start_workers'", "D:/ELK 8.10.2/logstash-8.10.2/logstash-core/lib/logstash/java_pipeline.rb:194:in `run'", "D:/ELK 8.10.2/logstash-8.10.2/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["D:/ELK 8.10.2/logstash-8.10.2/config/conf.d/bk_services/bkgetivrnumbersoffers_config.conf"], :thread=>"#"} [2023-12-25T09:41:48,671][INFO][logstash.javapipeline][bkgetivrnumbersoffers] Pipeline terminated {"pipeline.id"=>"bkgetivrnumbersoffers"} ``` The problem is, when i try to check status of a terminated pipeline, status is green ```auto PS C:\Users\ssabdelkader> Invoke-RestMethod -Method GET -Uri 'http://localhost:9600/_node/pipelines/bkgetivrnumbersoffers?pretty' host : SCPrWUWS05 version : 8.10.2 http_address : 127.0.0.1:9600 id : 1f4078e7-f92c-45a6-9366-956fe385d154 name : SCPrWUWS05 ephemeral_id : 24367b98-ee2d-4df5-a747-58df975fc306 status : green snapshot : False pipeline : @{workers=1; batch_size=125; batch_delay=50} pipelines : @{bkgetivrnumbersoffers=} ``` yet the query was not executed with the other running pipelines. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 22, 2024, 6:56am UTC](https://discuss.elastic.co/t/logstash-pipelines-not-visible-in-stack-monitoring/349894/6 "2024-01-22T06:56:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.