Logstash monitoring data not showing up in Kibana

[Pedro_Osorio (Pedro Osório)]

Hi all,

I've been trying to see Logstash metrics in Kibana, but with no success. My setup is as follows:

• metricbeat daemonset collecting host/pods metrics shipping to elastic
• metricbeat deployment collecting cluster/logstash metrics shipping to elastic
• logstash daemonset processing logs from several locations (filebeat) and shipping to elastic

I can see the data in the internal logstash monitoring index:

GET .monitoring-logstash-7*/_search

        "_index" : ".monitoring-logstash-7-mb-2022.02.10",
        "_type" : "_doc",
        "_id" : "BC_14H4BHBLacYYIGJle",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2022-02-10T00:07:44.887Z",
          "cluster_uuid" : "dev-ksfmdcnmfrg-logstash",
          "service" : {
            "address" : "http://logstash-logstash-headless:6060/_node/stats",
            "type" : "logstash"
          },
          "ecs" : {
            "version" : "1.12.0"
          },
          "agent" : {
            "version" : "7.16.3",
            "hostname" : "metricbeat-metricbeat-metrics-5dd6cd5dcf-k6z4j",
            "ephemeral_id" : "7347fe34-2784-4cb7-9aef-0ff65d5a17e8",
            "id" : "bb15c778-8ca9-41e8-813d-5523046f678f",
            "name" : "metricbeat-metricbeat-metrics-5dd6cd5dcf-k6z4j",
            "type" : "metricbeat"
          },
          "interval_ms" : 30000,
          "type" : "logstash_stats",
          "logstash_stats" : {
            "reloads" : {
              "failures" : 0,
              "successes" : 0
            },
            "os" : {
              "cpu" : {
                "load_average" : {
                  "15m" : 0.9,
                  "1m" : 0.49,
                  "5m" : 0.9
                }
              },
              "cgroup" : {
                "cpuacct" : {
                  "usage_nanos" : 1.7664837178667E13,
                  "control_group" : "/"
                },
                "cpu" : {
                  "stat" : {
                    "number_of_elapsed_periods" : 2008038.0,
                    "number_of_times_throttled" : 104308,
                    "time_throttled_nanos" : 5.276325872116E12
                  },
                  "control_group" : "/"
                }
              }
            },
            "pipelines" : [
              {
                "id" : "elastic-logstash-nginx",
                "hash" : "7e118f92188dceb6e8aa242b0c94420c95ae7c517e0d5795390ed1de0f66c2ae",
                "ephemeral_id" : "74f15250-5871-4ca9-ae9b-fe88a16eb908",
                "events" : {
                  "queue_push_duration_in_millis" : 375,
               ....

If I go to Stack Monitoring, Logstash doesn't show up. If i enter "Setup Mode" i do see the Logstash UI, but there is nothing there.

I'm running Kibana's monitoring UI is enabled and it is running as a super user at the moment, to rule out permissions.

Let me know if you can help.