Logstash Metrics unavailable on Kibana Stack Monitoring UI

Elastic Stack Logstash
1

Hi all

Have installed metricbeat to monitor Logstash Nodes. The data stream - .monitoring-logstash-8-mb does get created and am seeing the data in the discover tab. But in the Stack Monitoring page for some reason the data for Logstash does not show up.
Facing the same issue for Beats (filebeat) Monitoring too.

Have used a similar setup to monitor Kibana and ES nodes and those work fine and am able to see the stats on the monitoring page

Any guide on this would be very helpful

PS: Running version of the stack is 8.7.1

2

Do you have any errors in your Metricbeat logs for the instances that are monitoring Logstash?
It could be that it's failing to collect one of the metricsets.
In Discover, which metricsets can you see?

Since you're on a newer version of Kibana, you have access to the Stack Monitoring health API that can help us debug what data is available.

For both Logstash and the Beats, are they all connected to the same Elasticsearch cluster that you are monitoring?

3

@miltonhultgren

Don't see any errors in the metricbeat logs.

Below is how the config looks like, currently have disabled Beats but yes they are connected to the same ES cluster.

    - module: kibana
      metricsets:
        - stats
      period: 10s
      hosts: ["<host>"]
      username: ""
      password: ""
      xpack.enabled: true
    - module: elasticsearch
      period: 10s
      hosts: ["<host>"]
      username: ""
      password: ""
      xpack.enabled: true
    - module: logstash
      period: 10s
      metricsets:
        - node
        - node_stats
      hosts: ["<host>"]
      username: ""
      password: ""
      xpack.enabled: true

Below is one of the outputs:

{
  "_index": ".ds-.monitoring-logstash-8-mb-2023.09.19-000001",
  "_id": "giLIrIoB2hnO2QgUetY4",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2023-09-19T09:32:13.970Z",
    "event": {
      "dataset": "logstash.node.stats",
      "module": "logstash",
      "duration": 58322714
    },
    "logstash": {
      "node": {
        "stats": {
          "timestamp": "2023-09-19T09:32:14.028Z",
          "events": {
            "duration_in_millis": 1026,
            "in": 0,
            "filtered": 0,
            "out": 0
          },
          "reloads": {
            "failures": 0,
            "successes": 0
          },
          "queue": {
            "events_count": 0
          },
          "process": {
            "open_file_descriptors": 173,
            "max_file_descriptors": 1048576,
            "cpu": {
              "percent": 7
            }
          },
          "os": {
            "cpu": {
              "percent": 0,
              "load_average": {
                "15m": 5.96,
                "1m": 5.25,
                "5m": 6.66
              }
            },
            "cgroup": {
              "cpuacct": null,
              "cpu": {
                "control_group": "",
                "cfs_quota_micros": 0,
                "stat": null
              }
            }
          },
          "jvm": {
            "gc": {
              "collectors": {
                "old": {
                  "collection_count": 0,
                  "collection_time_in_millis": 0
                },
                "young": {
                  "collection_count": 220,
                  "collection_time_in_millis": 4605
                }
              }
            },
            "mem": {
              "heap_max_in_bytes": 4294967294,
              "heap_used_in_bytes": 2038912656,
              "heap_used_percent": 47
            },
            "uptime_in_millis": 8642993
          },
          "pipelines": [
            {
              "events": {
                "queue_push_duration_in_millis": 0,
                "duration_in_millis": 283,
                "filtered": 0,
                "in": 0,
                "out": 0
              },
              "reloads": {
                "successes": 0,
                "failures": 0
              },
              "queue": {
                "events_count": 0,
                "max_queue_size_in_bytes": 0,
                "queue_size_in_bytes": 0,
                "type": "memory"
              },
              "vertices": [
                {
                  "pipeline_ephemeral_id": "9a190692-8c9d-4d5b-b2df-f760f83e359a",
                  "queue_push_duration_in_millis": 0,
                  "events_out": 0,
                  "id": "kafka-nonprod"
                },
                {
                  "duration_in_millis": 81,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "173a762c4b4a8401cb50e7091c6372eb665d02155a5d80cd99c94cf34fe0fa22",
                  "pipeline_ephemeral_id": "9a190692-8c9d-4d5b-b2df-f760f83e359a"
                },
                {
                  "events_out": 0,
                  "id": "106c904cd1afa52d0d97fbbcad7cd087950303bad607fc67ad628cf91189595c",
                  "pipeline_ephemeral_id": "9a190692-8c9d-4d5b-b2df-f760f83e359a",
                  "duration_in_millis": 11,
                  "events_in": 0
                },
                {
                  "events_out": 0,
                  "id": "elastic-nonprod",
                  "pipeline_ephemeral_id": "9a190692-8c9d-4d5b-b2df-f760f83e359a",
                  "cluster_uuid": "MkUomKBLT6aa2MwBprBc_A",
                  "duration_in_millis": 98,
                  "events_in": 0
                }
              ],
              "id": "nonprodlogs",
              "hash": "2d28d16bdcbfc5043f824ef368c72c7add4206d00959628775d9e8a7da3c162f",
              "ephemeral_id": "9a190692-8c9d-4d5b-b2df-f760f83e359a"
            },
            {
              "hash": "b81067844b9bd7ce93a455f6a867314d0eafd4bf8a265f10fbbd6b49c842ee59",
              "ephemeral_id": "c01dfe30-1ff5-497e-bd12-dbc8fd555799",
              "events": {
                "duration_in_millis": 286,
                "filtered": 0,
                "in": 0,
                "out": 0,
                "queue_push_duration_in_millis": 0
              },
              "reloads": {
                "successes": 0,
                "failures": 0
              },
              "queue": {
                "type": "memory",
                "events_count": 0,
                "max_queue_size_in_bytes": 0,
                "queue_size_in_bytes": 0
              },
              "vertices": [
                {
                  "events_out": 0,
                  "id": "kafka-perf-nonprod",
                  "pipeline_ephemeral_id": "c01dfe30-1ff5-497e-bd12-dbc8fd555799",
                  "queue_push_duration_in_millis": 0
                },
                {
                  "id": "5cd697a78e4614ce8bad85163662df4f4b544b4f5b26f1f6b91b60aa4390171b",
                  "pipeline_ephemeral_id": "c01dfe30-1ff5-497e-bd12-dbc8fd555799",
                  "duration_in_millis": 68,
                  "events_in": 0,
                  "events_out": 0
                },
                {
                  "duration_in_millis": 11,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "17ffbaaf92f5cdc150b92f22896e720ad3dc533d904d53293634d4034547d52f",
                  "pipeline_ephemeral_id": "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
                },
                {
                  "cluster_uuid": "MkUomKBLT6aa2MwBprBc_A",
                  "duration_in_millis": 106,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "elastic-perf-nonprod",
                  "pipeline_ephemeral_id": "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
                }
              ],
              "id": "perfcluster"
            },
            {
              "id": "signetperf",
              "hash": "a8796c1de82a8534fca361a09b7f93ac23b690570ad9cf8faedff718d22fc420",
              "ephemeral_id": "aab1a03c-3984-4517-b30d-3539ba3c12b0",
              "events": {
                "filtered": 0,
                "in": 0,
                "out": 0,
                "queue_push_duration_in_millis": 0,
                "duration_in_millis": 174
              },
              "reloads": {
                "failures": 0,
                "successes": 0
              },
              "queue": {
                "type": "memory",
                "events_count": 0,
                "max_queue_size_in_bytes": 0,
                "queue_size_in_bytes": 0
              },
              "vertices": [
                {
                  "events_out": 0,
                  "id": "kafka-signet-perf-nonprod",
                  "pipeline_ephemeral_id": "aab1a03c-3984-4517-b30d-3539ba3c12b0",
                  "queue_push_duration_in_millis": 0
                },
                {
                  "pipeline_ephemeral_id": "aab1a03c-3984-4517-b30d-3539ba3c12b0",
                  "duration_in_millis": 36,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "3f0db88085ac75ecc9e76656a6b6e2e8f5396e391d933339b4cbf0b1499d5e43"
                },
                {
                  "pipeline_ephemeral_id": "aab1a03c-3984-4517-b30d-3539ba3c12b0",
                  "duration_in_millis": 12,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "3ae09666f2cf5668681d20b0957e99106f7eab583678d1894594291737bef1ac"
                },
                {
                  "cluster_uuid": "MkUomKBLT6aa2MwBprBc_A",
                  "duration_in_millis": 58,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "elastic-signet-perf-nonprod",
                  "pipeline_ephemeral_id": "aab1a03c-3984-4517-b30d-3539ba3c12b0"
                }
              ]
            },
            {
              "hash": "da032c952cfa48c81d95b7aefc4508453a3c8c593e05fa0619d901238bda6fae",
              "ephemeral_id": "a53d534e-e568-492d-a635-a065093a1be4",
              "events": {
                "in": 0,
                "out": 0,
                "queue_push_duration_in_millis": 0,
                "duration_in_millis": 264,
                "filtered": 0
              },
              "reloads": {
                "successes": 0,
                "failures": 0
              },
              "queue": {
                "events_count": 0,
                "max_queue_size_in_bytes": 0,
                "queue_size_in_bytes": 0,
                "type": "memory"
              },
              "vertices": [
                {
                  "events_out": 0,
                  "id": "kafka-filebeat-metrics-nonprod",
                  "pipeline_ephemeral_id": "a53d534e-e568-492d-a635-a065093a1be4",
                  "queue_push_duration_in_millis": 0
                },
                {
                  "cluster_uuid": "MkUomKBLT6aa2MwBprBc_A",
                  "duration_in_millis": 141,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "5458cde1d3492f585a53f8348912826bb43c5ec17de43d1b037c28feb995ee17",
                  "pipeline_ephemeral_id": "a53d534e-e568-492d-a635-a065093a1be4"
                },
                {
                  "pipeline_ephemeral_id": "a53d534e-e568-492d-a635-a065093a1be4",
                  "cluster_uuid": "MkUomKBLT6aa2MwBprBc_A",
                  "duration_in_millis": 23,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "ee71743edcff7959c96e78ca85788300924cb785f702453653e9ab60e23987fc"
                }
              ],
              "id": "filebeatmetrics"
            }
          ],
          "logstash": {
            "host": "logstash-0",
            "snapshot": false,
            "http_address": "0.0.0.0:9600",
            "pipeline": {
              "batch_size": 250,
              "workers": 4
            },
            "uuid": "d7d77c8b-9b20-4c7c-a454-d6516bc6c886",
            "ephemeral_id": "0c2aa487-acec-44fa-9637-b5650816c011",
            "name": "logstash-0",
            "version": "8.7.1",
            "status": "green"
          }
        }
      },
      "cluster": {
        "id": "nonprod-logs-cluster"
      },
      "elasticsearch": {
        "cluster": {
          "id": "nonprod-logs-cluster"
        }
      }
    },
    "ecs": {
      "version": "8.0.0"
    },
    "host": {
      "name": "metricbeat-8.7.1-7d4ffc985b-pb98d"
    },
    "agent": {
      "name": "metricbeat-8.7.1-7d4ffc985b-pb98d",
      "type": "metricbeat",
      "version": "8.7.1",
      "ephemeral_id": "d5918dc5-cb84-4d12-aca3-a086041d8b56",
      "id": "2ee5eb1d-1ef4-42c5-8e6a-a5cfdfb3a7c5"
    },
    "cloud": {
      "region": "westus2",
      "provider": "azure",
      "account": {},
      "instance": {
        "name": "aks-azpool3-35535046-vmss_4",
        "id": "8818c149-18cf-4a6f-83f0-155271ea3b86"
      },
      "machine": {
        "type": "Standard_D4s_v3"
      },
      "service": {
        "name": "Virtual Machines"
      }
    },
    "service": {
      "type": "logstash",
      "name": "logstash",
      "id": "",
      "hostname": "logstash-0",
      "version": "8.7.1",
      "address": "http://logstash-0:9600/_node/stats"
    },
    "metricset": {
      "name": "node_stats",
      "period": 10000
    }
  },
  "fields": {
    "logstash.node.stats.events.out": [
      0
    ],
    "logstash_stats.events.out": [
      0
    ],
    "logstash_stats.queue.events_count": [
      0
    ],
    "logstash.node.stats.jvm.mem.heap_used_percent": [
      47
    ],
    "logstash.cluster.id": [
      "nonprod-logs-cluster"
    ],
    "logstash.node.stats.timestamp": [
      "2023-09-19T09:32:14.028Z"
    ],
    "service.type": [
      "logstash"
    ],
    "logstash.node.stats.process.max_file_descriptors": [
      1048576
    ],
    "logstash.node.stats.events.filtered": [
      0
    ],
    "logstash.node.stats.logstash.pipeline.batch_size": [
      250
    ],
    "logstash.node.stats.logstash.http_address": [
      "0.0.0.0:9600"
    ],
    "agent.name": [
      "metricbeat-8.7.1-7d4ffc985b-pb98d"
    ],
    "host.name": [
      "metricbeat-8.7.1-7d4ffc985b-pb98d"
    ],
    "logstash.node.stats.process.cpu.percent": [
      7
    ],
    "logstash.node.stats.queue.events_count": [
      0
    ],
    "logstash_stats.jvm.mem.heap_max_in_bytes": [
      4294967294
    ],
    "logstash.node.stats.logstash.pipeline.workers": [
      4
    ],
    "logstash.node.stats.logstash.status": [
      "green"
    ],
    "logstash.node.stats.jvm.mem.heap_max_in_bytes": [
      4294967294
    ],
    "cloud.region": [
      "westus2"
    ],
    "logstash.node.stats.reloads.successes": [
      0
    ],
    "logstash.node.stats.reloads.failures": [
      0
    ],
    "logstash_stats.logstash.uuid": [
      "d7d77c8b-9b20-4c7c-a454-d6516bc6c886"
    ],
    "cloud.provider": [
      "azure"
    ],
    "cloud.machine.type": [
      "Standard_D4s_v3"
    ],
    "logstash_stats.events.in": [
      0
    ],
    "cluster_uuid": [
      "nonprod-logs-cluster"
    ],
    "agent.id": [
      "2ee5eb1d-1ef4-42c5-8e6a-a5cfdfb3a7c5"
    ],
    "cloud.service.name": [
      "Virtual Machines"
    ],
    "logstash.node.stats.logstash.version": [
      "8.7.1"
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "service.address": [
      "http://logstash-0:9600/_node/stats"
    ],
    "logstash.node.stats.process.open_file_descriptors": [
      173
    ],
    "logstash_stats.events.duration_in_millis": [
      1026
    ],
    "service.hostname": [
      "logstash-0"
    ],
    "agent.version": [
      "8.7.1"
    ],
    "logstash.node.stats.events.duration_in_millis": [
      1026
    ],
    "logstash_stats.jvm.mem.heap_used_in_bytes": [
      2038912656
    ],
    "logstash.node.stats.logstash.snapshot": [
      false
    ],
    "logstash.node.stats.logstash.ephemeral_id": [
      "0c2aa487-acec-44fa-9637-b5650816c011"
    ],
    "logstash.node.stats.jvm.gc.collectors.young.collection_time_in_millis": [
      4605
    ],
    "service.id": [
      ""
    ],
    "logstash_stats.process.cpu.percent": [
      7
    ],
    "logstash_stats.os.cpu.load_average.5m": [
      6.6601562
    ],
    "logstash.node.stats.os.cgroup.cpu.control_group": [
      ""
    ],
    "logstash.node.stats.os.cpu.percent": [
      0
    ],
    "cloud.instance.id": [
      "8818c149-18cf-4a6f-83f0-155271ea3b86"
    ],
    "agent.type": [
      "metricbeat"
    ],
    "logstash_stats.timestamp": [
      "2023-09-19T09:32:13.970Z"
    ],
    "logstash_stats.os.cpu.load_average.1m": [
      5.25
    ],
    "event.module": [
      "logstash"
    ],
    "logstash.node.stats.logstash.name": [
      "logstash-0"
    ],
    "logstash_stats.os.cpu.load_average.15m": [
      5.9609375
    ],
    "logstash.node.stats.jvm.mem.heap_used_in_bytes": [
      2038912656
    ],
    "logstash.node.stats.pipelines": [
      {
        "events.duration_in_millis": [
          283
        ],
        "reloads.failures": [
          0
        ],
        "vertices": [
          {
            "id": [
              "kafka-nonprod"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "9a190692-8c9d-4d5b-b2df-f760f83e359a"
            ],
            "queue_push_duration_in_millis": [
              0
            ]
          },
          {
            "duration_in_millis": [
              81
            ],
            "id": [
              "173a762c4b4a8401cb50e7091c6372eb665d02155a5d80cd99c94cf34fe0fa22"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "9a190692-8c9d-4d5b-b2df-f760f83e359a"
            ],
            "events_in": [
              0
            ]
          },
          {
            "duration_in_millis": [
              11
            ],
            "id": [
              "106c904cd1afa52d0d97fbbcad7cd087950303bad607fc67ad628cf91189595c"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "9a190692-8c9d-4d5b-b2df-f760f83e359a"
            ],
            "events_in": [
              0
            ]
          },
          {
            "events_out": [
              0
            ],
            "cluster_uuid": [
              "MkUomKBLT6aa2MwBprBc_A"
            ],
            "pipeline_ephemeral_id": [
              "9a190692-8c9d-4d5b-b2df-f760f83e359a"
            ],
            "events_in": [
              0
            ],
            "duration_in_millis": [
              98
            ],
            "id": [
              "elastic-nonprod"
            ]
          }
        ],
        "queue.events_count": [
          0
        ],
        "queue.queue_size_in_bytes": [
          0
        ],
        "queue.max_queue_size_in_bytes": [
          0
        ],
        "events.filtered": [
          0
        ],
        "queue.type": [
          "memory"
        ],
        "events.queue_push_duration_in_millis": [
          0
        ],
        "events.in": [
          0
        ],
        "reloads.successes": [
          0
        ],
        "id": [
          "nonprodlogs"
        ],
        "ephemeral_id": [
          "9a190692-8c9d-4d5b-b2df-f760f83e359a"
        ],
        "hash": [
          "2d28d16bdcbfc5043f824ef368c72c7add4206d00959628775d9e8a7da3c162f"
        ],
        "events.out": [
          0
        ]
      },
      {
        "events.duration_in_millis": [
          286
        ],
        "reloads.failures": [
          0
        ],
        "vertices": [
          {
            "id": [
              "kafka-perf-nonprod"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
            ],
            "queue_push_duration_in_millis": [
              0
            ]
          },
          {
            "duration_in_millis": [
              68
            ],
            "id": [
              "5cd697a78e4614ce8bad85163662df4f4b544b4f5b26f1f6b91b60aa4390171b"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
            ],
            "events_in": [
              0
            ]
          },
          {
            "duration_in_millis": [
              11
            ],
            "id": [
              "17ffbaaf92f5cdc150b92f22896e720ad3dc533d904d53293634d4034547d52f"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
            ],
            "events_in": [
              0
            ]
          },
          {
            "events_out": [
              0
            ],
            "cluster_uuid": [
              "MkUomKBLT6aa2MwBprBc_A"
            ],
            "pipeline_ephemeral_id": [
              "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
            ],
            "events_in": [
              0
            ],
            "duration_in_millis": [
              106
            ],
            "id": [
              "elastic-perf-nonprod"
            ]
          }
        ],
        "queue.events_count": [
          0
        ],
        "queue.queue_size_in_bytes": [
          0
        ],
        "queue.max_queue_size_in_bytes": [
          0
        ],
        "events.filtered": [
          0
        ],
        "queue.type": [
          "memory"
        ],
        "events.queue_push_duration_in_millis": [
          0
        ],
        "events.in": [
          0
        ],
        "reloads.successes": [
          0
        ],
        "id": [
          "perfcluster"
        ],
        "ephemeral_id": [
          "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
        ],
        "hash": [
          "b81067844b9bd7ce93a455f6a867314d0eafd4bf8a265f10fbbd6b49c842ee59"
        ],
        "events.out": [
          0
        ]
      },
      {
        "events.duration_in_millis": [
          174
        ],
        "reloads.failures": [
          0
        ],
        "vertices": [
          {
            "id": [
              "kafka-signet-perf-nonprod"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "aab1a03c-3984-4517-b30d-3539ba3c12b0"
            ],
            "queue_push_duration_in_millis": [
              0
            ]
          },
          {
            "duration_in_millis": [
              36
            ],
            "id": [
              "3f0db88085ac75ecc9e76656a6b6e2e8f5396e391d933339b4cbf0b1499d5e43"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "aab1a03c-3984-4517-b30d-3539ba3c12b0"
            ],
            "events_in": [
              0
            ]
          },
          {
            "duration_in_millis": [
              12
            ],
            "id": [
              "3ae09666f2cf5668681d20b0957e99106f7eab583678d1894594291737bef1ac"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "aab1a03c-3984-4517-b30d-3539ba3c12b0"
            ],
            "events_in": [
              0
            ]
          },
          {
            "events_out": [
              0
            ],
            "cluster_uuid": [
              "MkUomKBLT6aa2MwBprBc_A"
            ],
            "pipeline_ephemeral_id": [
              "aab1a03c-3984-4517-b30d-3539ba3c12b0"
            ],
            "events_in": [
              0
            ],
            "duration_in_millis": [
              58
            ],
            "id": [
              "elastic-signet-perf-nonprod"
            ]
          }
        ],
        "queue.events_count": [
          0
        ],
        "queue.queue_size_in_bytes": [
          0
        ],
        "queue.max_queue_size_in_bytes": [
          0
        ],
        "events.filtered": [
          0
        ],
        "queue.type": [
          "memory"
        ],
        "events.queue_push_duration_in_millis": [
          0
        ],
        "events.in": [
          0
        ],
        "reloads.successes": [
          0
        ],
        "id": [
          "signetperf"
        ],
        "ephemeral_id": [
          "aab1a03c-3984-4517-b30d-3539ba3c12b0"
        ],
        "hash": [
          "a8796c1de82a8534fca361a09b7f93ac23b690570ad9cf8faedff718d22fc420"
        ],
        "events.out": [
          0
        ]
      },
      {
        "events.duration_in_millis": [
          264
        ],
        "reloads.failures": [
          0
        ],
        "vertices": [
          {
            "id": [
              "kafka-filebeat-metrics-nonprod"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "a53d534e-e568-492d-a635-a065093a1be4"
            ],
            "queue_push_duration_in_millis": [
              0
            ]
          },
          {
            "events_out": [
              0
            ],
            "cluster_uuid": [
              "MkUomKBLT6aa2MwBprBc_A"
            ],
            "pipeline_ephemeral_id": [
              "a53d534e-e568-492d-a635-a065093a1be4"
            ],
            "events_in": [
              0
            ],
            "duration_in_millis": [
              141
            ],
            "id": [
              "5458cde1d3492f585a53f8348912826bb43c5ec17de43d1b037c28feb995ee17"
            ]
          },
          {
            "events_out": [
              0
            ],
            "cluster_uuid": [
              "MkUomKBLT6aa2MwBprBc_A"
            ],
            "pipeline_ephemeral_id": [
              "a53d534e-e568-492d-a635-a065093a1be4"
            ],
            "events_in": [
              0
            ],
            "duration_in_millis": [
              23
            ],
            "id": [
              "ee71743edcff7959c96e78ca85788300924cb785f702453653e9ab60e23987fc"
            ]
          }
        ],
        "queue.events_count": [
          0
        ],
        "queue.queue_size_in_bytes": [
          0
        ],
        "queue.max_queue_size_in_bytes": [
          0
        ],
        "events.filtered": [
          0
        ],
        "queue.type": [
          "memory"
        ],
        "events.queue_push_duration_in_millis": [
          0
        ],
        "events.in": [
          0
        ],
        "reloads.successes": [
          0
        ],
        "id": [
          "filebeatmetrics"
        ],
        "ephemeral_id": [
          "a53d534e-e568-492d-a635-a065093a1be4"
        ],
        "hash": [
          "da032c952cfa48c81d95b7aefc4508453a3c8c593e05fa0619d901238bda6fae"
        ],
        "events.out": [
          0
        ]
      }
    ],
    "timestamp": [
      "2023-09-19T09:32:13.970Z"
    ],
    "logstash_stats.logstash.version": [
      "8.7.1"
    ],
    "logstash.node.stats.os.cpu.load_average.15m": [
      5.9609375
    ],
    "service.name": [
      "logstash"
    ],
    "logstash.node.stats.jvm.gc.collectors.old.collection_count": [
      0
    ],
    "logstash.node.stats.jvm.gc.collectors.old.collection_time_in_millis": [
      0
    ],
    "logstash.node.stats.jvm.gc.collectors.young.collection_count": [
      220
    ],
    "metricset.period": [
      10000
    ],
    "logstash_stats.os.cgroup.cpu.cfs_quota_micros": [
      0
    ],
    "logstash.node.stats.logstash.host": [
      "logstash-0"
    ],
    "logstash.node.stats.jvm.uptime_in_millis": [
      8642993
    ],
    "logstash.node.stats.events.in": [
      0
    ],
    "logstash.node.stats.logstash.uuid": [
      "d7d77c8b-9b20-4c7c-a454-d6516bc6c886"
    ],
    "metricset.name": [
      "node_stats"
    ],
    "event.duration": [
      58322714
    ],
    "logstash.node.stats.os.cpu.load_average.1m": [
      5.25
    ],
    "@timestamp": [
      "2023-09-19T09:32:13.970Z"
    ],
    "logstash.node.stats.os.cgroup.cpu.cfs_quota_micros": [
      0
    ],
    "service.version": [
      "8.7.1"
    ],
    "logstash.node.stats.os.cpu.load_average.5m": [
      6.6601562
    ],
    "agent.ephemeral_id": [
      "d5918dc5-cb84-4d12-aca3-a086041d8b56"
    ],
    "logstash_stats.jvm.uptime_in_millis": [
      8642993
    ],
    "event.dataset": [
      "logstash.node.stats"
    ],
    "logstash.elasticsearch.cluster.id": [
      "nonprod-logs-cluster"
    ],
    "cloud.instance.name": [
      "aks-azpool3-35535046-vmss_4"
    ]
  }
}
4

Did you specify this as the cluster ID in the logstash.yml?
Is there a matching Elasticsearch cluster for this?

I think the Stack Monitoring UI won't know where to display that data otherwise.
What about the result from the Health API?

5

Thanks @miltonhultgren that gave me an idea, actually had set monitoring.cluster_uuid: "nonprod-logs-cluster" but then realized the actual UUID was different a randomly generated id.

6

For Beats the approach we follow is push the output to kafka -> logstash and then read it into ES. The datastream that get's created is not of the format .monitoring-beats-8-mb. But of the format metricbeat-*. Will the Stack Monitoring still be able to display the data?

7

Right, so for Logstash, please align the ID with the randomly generated ID of the Elasticsearch cluster.

For the Beats data, it has to end up in .monitoring-* for the Stack Monitoring UI to use it, so you'll need to figure out how to route the data to the right data stream. I'm not sure how that is done, what does your Logstash output look like today?

8

Got it thanks again, for beats we have just routed the logs directly to ES for now and things seem fine.