As per the compatibility matrix, Logstash 5.6.16 is supported with ES and Kibana 6.8.6. However, in my case, while Logstash 5.6.16 is able to push monitoring data to ES 6.8.6, Logstash monitoring details are NOT visible in Kibana UI.
Steps to Reproduce:
1. Install and set-up ES 6.8.6.
2. Install and set-up Kibana 6.8.6
3. Ensure you are able to see monitoring details of Kibana and ES (you should be able to)
4. Now install logstash 5.6.16.
5. Start logstash with a v basic ".conf" file.
6. It will **not** be visible in monitoring
7. Now install Logstash 6.8.6 and start with v simple ".conf" file. it will be *visible* in monitoring.
8. Now you will also be able to see logstash 5.6.16 as well in monitoring.
From logstash logs:
[2020-04-08T06:48:44,650][INFO ][logstash.pipeline ] Starting pipeline {"id"=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2}
...
[2020-04-08T06:48:44,949][INFO ][logstash.pipeline ] Pipeline .monitoring-logstash started
...
[2020-04-08T06:48:46,252][INFO ][logstash.pipeline ] Pipeline main started
...
[2020-04-08T06:48:54,955][INFO ][logstash.inputs.metrics ] Monitoring License OK
The logstash indices are created fine in Elasticsearch:
green open .monitoring-logstash-6-2020.04.03 1 0 66151 0 20.8mb 20.8mb
green open .monitoring-logstash-6-2020.04.04 1 0 60468 0 18.5mb 18.5mb
green open .monitoring-logstash-6-2020.04.05 1 0 60977 0 18.7mb 18.7mb
green open .monitoring-logstash-6-2020.04.06 1 0 69096 0 21.6mb 21.6mb
green open .monitoring-logstash-6-2020.04.07 1 0 80621 0 26mb 26mb
green open .monitoring-logstash-6-2020.04.08 1 0 86386 0 26.7mb 26.7mb
green open .monitoring-logstash-6-2020.04.09 1 0 47210 0 14.7mb 14.7mb
A document from today's logstash monitoring index where you can see the version is 5.6.16.
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 47312,
"max_score" : null,
"hits" : [
{
"_index" : ".monitoring-logstash-6-2020.04.09",
"_type" : "doc",
"_id" : "y2s8XHEB_tpMl4SDfR6u",
"_score" : null,
"_source" : {
"cluster_uuid" : "<redacted_UUID>",
"timestamp" : "2020-04-09T00:00:00.252Z",
"interval_ms" : 1000,
"type" : "logstash_stats",
"source_node" : {
"uuid" : "<UUID>",
"host" : "1.1.1.1",
"transport_address" : "1.1.1.1:9300",
"ip" : "1.1.1.1",
"name" : "foo.bar.com",
"timestamp" : "2020-04-09T00:00:00.252Z"
},
"logstash_stats" : {
"jvm" : {
"uptime_in_millis" : 145873108,
"mem" : {
"heap_used_in_bytes" : 291063208,
"heap_used_percent" : 27,
"heap_max_in_bytes" : 1056309248
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 145369,
"collection_time_in_millis" : 1348649
},
"old" : {
"collection_count" : 715,
"collection_time_in_millis" : 53274
}
}
}
},
"logstash" : {
"pipeline" : {
"workers" : 4,
"batch_size" : 500
},
"http_address" : "127.0.0.1:9600",
"name" : "foobar",
"host" : "foobar",
"uuid" : "long-UUID",
"version" : "5.6.16",
"snapshot" : false,
"status" : "green"
}
Logstash.yml:
---
path.data: "/var/lib/logstash"
path.config: "/etc/logstash/conf.d"
path.logs: "/var/log/logstash"
xpack.monitoring.elasticsearch.url:
- http://foo.bar1.com:9200
- http://foo.bar2.com:9200
- http://foo.bar3.com:9200
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: <foo_pass>
xpack.monitoring.enabled: true
pipeline.batch.size: 250
pipeline.workers: 2