Logstash monitoring not showing in Kibana

Elastic Stack Logstash
1

Hello, I'm setting up a brand new ELK stack. All nodes are on 7.4.0. Xpack monitoring is turned on and I can see metrics for Elasticsearch and Kibana with no issues. However, nothing is showing for my Logstash nodes.

It's interesting because I see .monitoring-logstash-7-* indexes made from yesterday (10.01) but not today. I tried various troubleshooting steps and tried searching for solutions on the forums. Hopefully someone can help me find a way to solve this.

Here is my logstash.yml:

http.host: 10.220.0.252
http.port: 9600

path.logs: /var/log/logstash
# path.data: /var/lib/logstash

config.debug: true
log.level: debug

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["https://10.220.0.184:9200", "https://10.220.10.42:9200", "https://10.220.20.91:9200"]
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/certs/security-logging-elk-ca.crt
xpack.monitoring.elasticsearch.ssl.verification_mode: none
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: REMOVED
xpack.monitoring.collection.pipeline.details.enabled: true

I'll post my logs in a reply since I hit the limit when pasting.

2

And here are some of my logs during startup:

[2019-10-02T21:06:23,704][INFO ][logstash.javapipeline    ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, :thread=>"#<Thread:0x3a83c560 run>"}
[2019-10-02T21:06:23,762][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2019-10-02T21:06:23,773][DEBUG][logstash.javapipeline    ] Pipeline started successfully {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x3a83c560 run>"}
[2019-10-02T21:06:23,787][DEBUG][org.logstash.execution.PeriodicFlush] Pushing flush onto pipeline.
[2019-10-02T21:06:23,810][DEBUG][logstash.inputs.metrics  ][.monitoring-logstash] Metric: input started
[2019-10-02T21:06:23,886][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:".monitoring-logstash"], :non_running_pipelines=>[]}
[2019-10-02T21:06:23,939][DEBUG][logstash.agent           ] Starting puma
[2019-10-02T21:06:23,966][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2019-10-02T21:06:23,993][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2019-10-02T21:06:23,995][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2019-10-02T21:06:23,996][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2019-10-02T21:06:24,007][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2019-10-02T21:06:24,012][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>1}
[2019-10-02T21:06:24,023][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>1}
[2019-10-02T21:06:24,034][DEBUG][logstash.agent           ] Executing action {:action=>LogStash::PipelineAction::Stop/pipeline_id:.monitoring-logstash}
[2019-10-02T21:06:24,064][DEBUG][logstash.javapipeline    ] Closing inputs {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x3a83c560 sleep>"}
[2019-10-02T21:06:24,067][DEBUG][logstash.inputs.metrics  ] Stopping {:plugin=>"LogStash::Inputs::Metrics"}
[2019-10-02T21:06:24,073][DEBUG][logstash.inputs.metrics  ] Metrics input: stopped
[2019-10-02T21:06:24,092][DEBUG][logstash.javapipeline    ] Closed inputs {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x3a83c560 sleep>"}
[2019-10-02T21:06:24,106][DEBUG][logstash.javapipeline    ] Closing inputs {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x3a83c560 sleep>"}
[2019-10-02T21:06:24,119][DEBUG][logstash.api.service     ] [api-service] start
[2019-10-02T21:06:24,416][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2019-10-02T21:06:23,786][DEBUG][org.logstash.config.ir.CompiledPipeline][.monitoring-logstash] Compiled output
 P[output-elasticsearch{"hosts"=>["https://10.220.0.184:9200", "https://10.220.10.42:9200", "https://10.220.20.91:9200"], "bulk_path"=>"/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", "manage_template"=>"false", "document_type"=>"%{[@metadata][document_type]}", "index"=>"", "sniffing"=>"false", "user"=>"logstash_system", "password"=>"REMOVED", "ssl"=>"true", "cacert"=>"/etc/logstash/certs/security-logging-elk-ca.crt", "ssl_certificate_verification"=>"false"}|[str]pipeline:14:3:```
elasticsearch {
    hosts => ["https://10.220.0.184:9200", "https://10.220.10.42:9200", "https://10.220.20.91:9200"]
    bulk_path => "/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s"
    manage_template => false
    document_type => "%{[@metadata][document_type]}"
    index => ""
    sniffing => false

    user => "logstash_system"
    password => "REMOVED"


    ssl => true

        cacert => "/etc/logstash/certs/security-logging-elk-ca.crt"



    ssl_certificate_verification => false

  }
```]
 into
 org.logstash.config.ir.compiler.ComputeStepSyntaxElement@c96a6eaa
[2019-10-02T21:06:24,876][DEBUG][logstash.javapipeline    ][.monitoring-logstash] Input plugins stopped! Will shutdown filter/output workers. {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x3a83c560 run>"}
[2019-10-02T21:06:24,886][DEBUG][logstash.javapipeline    ][.monitoring-logstash] Shutdown waiting for worker thread {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x56160e1d run>"}
[2019-10-02T21:06:24,991][DEBUG][logstash.javapipeline    ] Worker closed {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x3a83c560 run>"}
[2019-10-02T21:06:24,994][DEBUG][logstash.outputs.elasticsearch][.monitoring-logstash] Closing {:plugin=>"LogStash::Outputs::ElasticSearch"}
[2019-10-02T21:06:25,006][DEBUG][logstash.outputs.elasticsearch][.monitoring-logstash] Stopping sniffer
[2019-10-02T21:06:25,009][DEBUG][logstash.outputs.elasticsearch][.monitoring-logstash] Stopping resurrectionist
[2019-10-02T21:06:25,022][INFO ][logstash.javapipeline    ] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[2019-10-02T21:06:25,578][DEBUG][logstash.outputs.elasticsearch][.monitoring-logstash] Waiting for in use manticore connections
[2019-10-02T21:06:25,586][DEBUG][logstash.outputs.elasticsearch][.monitoring-logstash] Closing adapter #<LogStash::Outputs::ElasticSearch::HttpClient::ManticoreAdapter:0x5b9f4be4>
[2019-10-02T21:06:25,594][DEBUG][logstash.pluginmetadata  ][.monitoring-logstash] Removing metadata for plugin 92b290db4bb323cb508e739410f00cb16d634c0b6d0f89be02a5e5afe282fddf
[2019-10-02T21:06:25,598][DEBUG][logstash.javapipeline    ][.monitoring-logstash] Pipeline has been shutdown {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x3a83c560 run>"}
[2019-10-02T21:06:25,651][INFO ][logstash.runner          ] Logstash shut down.

Seems like the monitoring pipeline starts and stops immediately. Is this normal?

I also don't have any of my own pipelines setup just yet. I want to make sure monitoring is started so I can use the pipeline viewer in Kibana.

3

That's interesting indeed. Do you recall doing something different with the Logstash configuration or startup yesterday vs. today? Did you have any pipelines running yesterday?

Also, what command are you using to start up Logstash?

Thanks,

Shaunak

4

I think that might be the problem. If there is nothing to monitor then monitoring shuts itself down. Try adding a simple pipeline to your configuration.

5

@shaunak The other day I did enable Centralized Pipeline Management, but I turned it off to troubleshoot just the Logstash monitoring issue.

I just re-enabled it and the current monitor index are being made now! This goes with what @Badger mentioned.

Once I defined the Pipelines in my logstash.yml and matched it with a Pipeline created in Kibana, the monitor index was created and also Logstash is now showing in the monitoring tab in Kibana.

I wish this was mentioned in the Logstash documents that you should have a Pipeline created for monitoring to work.

1 Like
6

I agree! If you have a GitHub account, please feel free to file na issue for this request here: Sign in to GitHub · GitHub. That way, you'll be able to track its progress and engage in discussion. If you don't have a GitHub account, let me know and I'll file the issue on your behalf. Thanks!

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.