Logstash Metrics unavailable on Kibana Stack Monitoring UI

Elastic Stack Logstash
1

Hi all

Have installed metricbeat to monitor Logstash Nodes. The data stream - .monitoring-logstash-8-mb does get created and am seeing the data in the discover tab. But in the Stack Monitoring page for some reason the data for Logstash does not show up.
Facing the same issue for Beats (filebeat) Monitoring too.

Have used a similar setup to monitor Kibana and ES nodes and those work fine and am able to see the stats on the monitoring page

Any guide on this would be very helpful

PS: Running version of the stack is 8.7.1

2

Do you have any errors in your Metricbeat logs for the instances that are monitoring Logstash?
It could be that it's failing to collect one of the metricsets.
In Discover, which metricsets can you see?

Since you're on a newer version of Kibana, you have access to the Stack Monitoring health API that can help us debug what data is available.

For both Logstash and the Beats, are they all connected to the same Elasticsearch cluster that you are monitoring?

3

@miltonhultgren

Don't see any errors in the metricbeat logs.

Below is how the config looks like, currently have disabled Beats but yes they are connected to the same ES cluster.

    - module: kibana
      metricsets:
        - stats
      period: 10s
      hosts: ["<host>"]
      username: ""
      password: ""
      xpack.enabled: true
    - module: elasticsearch
      period: 10s
      hosts: ["<host>"]
      username: ""
      password: ""
      xpack.enabled: true
    - module: logstash
      period: 10s
      metricsets:
        - node
        - node_stats
      hosts: ["<host>"]
      username: ""
      password: ""
      xpack.enabled: true

Below is one of the outputs:

{
  "_index": ".ds-.monitoring-logstash-8-mb-2023.09.19-000001",
  "_id": "giLIrIoB2hnO2QgUetY4",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2023-09-19T09:32:13.970Z",
    "event": {
      "dataset": "logstash.node.stats",
      "module": "logstash",
      "duration": 58322714
    },
    "logstash": {
      "node": {
        "stats": {
          "timestamp": "2023-09-19T09:32:14.028Z",
          "events": {
            "duration_in_millis": 1026,
            "in": 0,
            "filtered": 0,
            "out": 0
          },
          "reloads": {
            "failures": 0,
            "successes": 0
          },
          "queue": {
            "events_count": 0
          },
          "process": {
            "open_file_descriptors": 173,
            "max_file_descriptors": 1048576,
            "cpu": {
              "percent": 7
            }
          },
          "os": {
            "cpu": {
              "percent": 0,
              "load_average": {
                "15m": 5.96,
                "1m": 5.25,
                "5m": 6.66
              }
            },
            "cgroup": {
              "cpuacct": null,
              "cpu": {
                "control_group": "",
                "cfs_quota_micros": 0,
                "stat": null
              }
            }
          },
          "jvm": {
            "gc": {
              "collectors": {
                "old": {
                  "collection_count": 0,
                  "collection_time_in_millis": 0
                },
                "young": {
                  "collection_count": 220,
                  "collection_time_in_millis": 4605
                }
              }
            },
            "mem": {
              "heap_max_in_bytes": 4294967294,
              "heap_used_in_bytes": 2038912656,
              "heap_used_percent": 47
            },
            "uptime_in_millis": 8642993
          },
          "pipelines": [
            {
              "events": {
                "queue_push_duration_in_millis": 0,
                "duration_in_millis": 283,
                "filtered": 0,
                "in": 0,
                "out": 0
              },
              "reloads": {
                "successes": 0,
                "failures": 0
              },
              "queue": {
                "events_count": 0,
                "max_queue_size_in_bytes": 0,
                "queue_size_in_bytes": 0,
                "type": "memory"
              },
              "vertices": [
                {
                  "pipeline_ephemeral_id": "9a190692-8c9d-4d5b-b2df-f760f83e359a",
                  "queue_push_duration_in_millis": 0,
                  "events_out": 0,
                  "id": "kafka-nonprod"
                },
                {
                  "duration_in_millis": 81,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "173a762c4b4a8401cb50e7091c6372eb665d02155a5d80cd99c94cf34fe0fa22",
                  "pipeline_ephemeral_id": "9a190692-8c9d-4d5b-b2df-f760f83e359a"
                },
                {
                  "events_out": 0,
                  "id": "106c904cd1afa52d0d97fbbcad7cd087950303bad607fc67ad628cf91189595c",
                  "pipeline_ephemeral_id": "9a190692-8c9d-4d5b-b2df-f760f83e359a",
                  "duration_in_millis": 11,
                  "events_in": 0
                },
                {
                  "events_out": 0,
                  "id": "elastic-nonprod",
                  "pipeline_ephemeral_id": "9a190692-8c9d-4d5b-b2df-f760f83e359a",
                  "cluster_uuid": "MkUomKBLT6aa2MwBprBc_A",
                  "duration_in_millis": 98,
                  "events_in": 0
                }
              ],
              "id": "nonprodlogs",
              "hash": "2d28d16bdcbfc5043f824ef368c72c7add4206d00959628775d9e8a7da3c162f",
              "ephemeral_id": "9a190692-8c9d-4d5b-b2df-f760f83e359a"
            },
            {
              "hash": "b81067844b9bd7ce93a455f6a867314d0eafd4bf8a265f10fbbd6b49c842ee59",
              "ephemeral_id": "c01dfe30-1ff5-497e-bd12-dbc8fd555799",
              "events": {
                "duration_in_millis": 286,
                "filtered": 0,
                "in": 0,
                "out": 0,
                "queue_push_duration_in_millis": 0
              },
              "reloads": {
                "successes": 0,
                "failures": 0
              },
              "queue": {
                "type": "memory",
                "events_count": 0,
                "max_queue_size_in_bytes": 0,
                "queue_size_in_bytes": 0
              },
              "vertices": [
                {
                  "events_out": 0,
                  "id": "kafka-perf-nonprod",
                  "pipeline_ephemeral_id": "c01dfe30-1ff5-497e-bd12-dbc8fd555799",
                  "queue_push_duration_in_millis": 0
                },
                {
                  "id": "5cd697a78e4614ce8bad85163662df4f4b544b4f5b26f1f6b91b60aa4390171b",
                  "pipeline_ephemeral_id": "c01dfe30-1ff5-497e-bd12-dbc8fd555799",
                  "duration_in_millis": 68,
                  "events_in": 0,
                  "events_out": 0
                },
                {
                  "duration_in_millis": 11,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "17ffbaaf92f5cdc150b92f22896e720ad3dc533d904d53293634d4034547d52f",
                  "pipeline_ephemeral_id": "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
                },
                {
                  "cluster_uuid": "MkUomKBLT6aa2MwBprBc_A",
                  "duration_in_millis": 106,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "elastic-perf-nonprod",
                  "pipeline_ephemeral_id": "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
                }
              ],
              "id": "perfcluster"
            },
            {
              "id": "signetperf",
              "hash": "a8796c1de82a8534fca361a09b7f93ac23b690570ad9cf8faedff718d22fc420",
              "ephemeral_id": "aab1a03c-3984-4517-b30d-3539ba3c12b0",
              "events": {
                "filtered": 0,
                "in": 0,
                "out": 0,
                "queue_push_duration_in_millis": 0,
                "duration_in_millis": 174
              },
              "reloads": {
                "failures": 0,
                "successes": 0
              },
              "queue": {
                "type": "memory",
                "events_count": 0,
                "max_queue_size_in_bytes": 0,
                "queue_size_in_bytes": 0
              },
              "vertices": [
                {
                  "events_out": 0,
                  "id": "kafka-signet-perf-nonprod",
                  "pipeline_ephemeral_id": "aab1a03c-3984-4517-b30d-3539ba3c12b0",
                  "queue_push_duration_in_millis": 0
                },
                {
                  "pipeline_ephemeral_id": "aab1a03c-3984-4517-b30d-3539ba3c12b0",
                  "duration_in_millis": 36,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "3f0db88085ac75ecc9e76656a6b6e2e8f5396e391d933339b4cbf0b1499d5e43"
                },
                {
                  "pipeline_ephemeral_id": "aab1a03c-3984-4517-b30d-3539ba3c12b0",
                  "duration_in_millis": 12,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "3ae09666f2cf5668681d20b0957e99106f7eab583678d1894594291737bef1ac"
                },
                {
                  "cluster_uuid": "MkUomKBLT6aa2MwBprBc_A",
                  "duration_in_millis": 58,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "elastic-signet-perf-nonprod",
                  "pipeline_ephemeral_id": "aab1a03c-3984-4517-b30d-3539ba3c12b0"
                }
              ]
            },
            {
              "hash": "da032c952cfa48c81d95b7aefc4508453a3c8c593e05fa0619d901238bda6fae",
              "ephemeral_id": "a53d534e-e568-492d-a635-a065093a1be4",
              "events": {
                "in": 0,
                "out": 0,
                "queue_push_duration_in_millis": 0,
                "duration_in_millis": 264,
                "filtered": 0
              },
              "reloads": {
                "successes": 0,
                "failures": 0
              },
              "queue": {
                "events_count": 0,
                "max_queue_size_in_bytes": 0,
                "queue_size_in_bytes": 0,
                "type": "memory"
              },
              "vertices": [
                {
                  "events_out": 0,
                  "id": "kafka-filebeat-metrics-nonprod",
                  "pipeline_ephemeral_id": "a53d534e-e568-492d-a635-a065093a1be4",
                  "queue_push_duration_in_millis": 0
                },
                {
                  "cluster_uuid": "MkUomKBLT6aa2MwBprBc_A",
                  "duration_in_millis": 141,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "5458cde1d3492f585a53f8348912826bb43c5ec17de43d1b037c28feb995ee17",
                  "pipeline_ephemeral_id": "a53d534e-e568-492d-a635-a065093a1be4"
                },
                {
                  "pipeline_ephemeral_id": "a53d534e-e568-492d-a635-a065093a1be4",
                  "cluster_uuid": "MkUomKBLT6aa2MwBprBc_A",
                  "duration_in_millis": 23,
                  "events_in": 0,
                  "events_out": 0,
                  "id": "ee71743edcff7959c96e78ca85788300924cb785f702453653e9ab60e23987fc"
                }
              ],
              "id": "filebeatmetrics"
            }
          ],
          "logstash": {
            "host": "logstash-0",
            "snapshot": false,
            "http_address": "0.0.0.0:9600",
            "pipeline": {
              "batch_size": 250,
              "workers": 4
            },
            "uuid": "d7d77c8b-9b20-4c7c-a454-d6516bc6c886",
            "ephemeral_id": "0c2aa487-acec-44fa-9637-b5650816c011",
            "name": "logstash-0",
            "version": "8.7.1",
            "status": "green"
          }
        }
      },
      "cluster": {
        "id": "nonprod-logs-cluster"
      },
      "elasticsearch": {
        "cluster": {
          "id": "nonprod-logs-cluster"
        }
      }
    },
    "ecs": {
      "version": "8.0.0"
    },
    "host": {
      "name": "metricbeat-8.7.1-7d4ffc985b-pb98d"
    },
    "agent": {
      "name": "metricbeat-8.7.1-7d4ffc985b-pb98d",
      "type": "metricbeat",
      "version": "8.7.1",
      "ephemeral_id": "d5918dc5-cb84-4d12-aca3-a086041d8b56",
      "id": "2ee5eb1d-1ef4-42c5-8e6a-a5cfdfb3a7c5"
    },
    "cloud": {
      "region": "westus2",
      "provider": "azure",
      "account": {},
      "instance": {
        "name": "aks-azpool3-35535046-vmss_4",
        "id": "8818c149-18cf-4a6f-83f0-155271ea3b86"
      },
      "machine": {
        "type": "Standard_D4s_v3"
      },
      "service": {
        "name": "Virtual Machines"
      }
    },
    "service": {
      "type": "logstash",
      "name": "logstash",
      "id": "",
      "hostname": "logstash-0",
      "version": "8.7.1",
      "address": "http://logstash-0:9600/_node/stats"
    },
    "metricset": {
      "name": "node_stats",
      "period": 10000
    }
  },
  "fields": {
    "logstash.node.stats.events.out": [
      0
    ],
    "logstash_stats.events.out": [
      0
    ],
    "logstash_stats.queue.events_count": [
      0
    ],
    "logstash.node.stats.jvm.mem.heap_used_percent": [
      47
    ],
    "logstash.cluster.id": [
      "nonprod-logs-cluster"
    ],
    "logstash.node.stats.timestamp": [
      "2023-09-19T09:32:14.028Z"
    ],
    "service.type": [
      "logstash"
    ],
    "logstash.node.stats.process.max_file_descriptors": [
      1048576
    ],
    "logstash.node.stats.events.filtered": [
      0
    ],
    "logstash.node.stats.logstash.pipeline.batch_size": [
      250
    ],
    "logstash.node.stats.logstash.http_address": [
      "0.0.0.0:9600"
    ],
    "agent.name": [
      "metricbeat-8.7.1-7d4ffc985b-pb98d"
    ],
    "host.name": [
      "metricbeat-8.7.1-7d4ffc985b-pb98d"
    ],
    "logstash.node.stats.process.cpu.percent": [
      7
    ],
    "logstash.node.stats.queue.events_count": [
      0
    ],
    "logstash_stats.jvm.mem.heap_max_in_bytes": [
      4294967294
    ],
    "logstash.node.stats.logstash.pipeline.workers": [
      4
    ],
    "logstash.node.stats.logstash.status": [
      "green"
    ],
    "logstash.node.stats.jvm.mem.heap_max_in_bytes": [
      4294967294
    ],
    "cloud.region": [
      "westus2"
    ],
    "logstash.node.stats.reloads.successes": [
      0
    ],
    "logstash.node.stats.reloads.failures": [
      0
    ],
    "logstash_stats.logstash.uuid": [
      "d7d77c8b-9b20-4c7c-a454-d6516bc6c886"
    ],
    "cloud.provider": [
      "azure"
    ],
    "cloud.machine.type": [
      "Standard_D4s_v3"
    ],
    "logstash_stats.events.in": [
      0
    ],
    "cluster_uuid": [
      "nonprod-logs-cluster"
    ],
    "agent.id": [
      "2ee5eb1d-1ef4-42c5-8e6a-a5cfdfb3a7c5"
    ],
    "cloud.service.name": [
      "Virtual Machines"
    ],
    "logstash.node.stats.logstash.version": [
      "8.7.1"
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "service.address": [
      "http://logstash-0:9600/_node/stats"
    ],
    "logstash.node.stats.process.open_file_descriptors": [
      173
    ],
    "logstash_stats.events.duration_in_millis": [
      1026
    ],
    "service.hostname": [
      "logstash-0"
    ],
    "agent.version": [
      "8.7.1"
    ],
    "logstash.node.stats.events.duration_in_millis": [
      1026
    ],
    "logstash_stats.jvm.mem.heap_used_in_bytes": [
      2038912656
    ],
    "logstash.node.stats.logstash.snapshot": [
      false
    ],
    "logstash.node.stats.logstash.ephemeral_id": [
      "0c2aa487-acec-44fa-9637-b5650816c011"
    ],
    "logstash.node.stats.jvm.gc.collectors.young.collection_time_in_millis": [
      4605
    ],
    "service.id": [
      ""
    ],
    "logstash_stats.process.cpu.percent": [
      7
    ],
    "logstash_stats.os.cpu.load_average.5m": [
      6.6601562
    ],
    "logstash.node.stats.os.cgroup.cpu.control_group": [
      ""
    ],
    "logstash.node.stats.os.cpu.percent": [
      0
    ],
    "cloud.instance.id": [
      "8818c149-18cf-4a6f-83f0-155271ea3b86"
    ],
    "agent.type": [
      "metricbeat"
    ],
    "logstash_stats.timestamp": [
      "2023-09-19T09:32:13.970Z"
    ],
    "logstash_stats.os.cpu.load_average.1m": [
      5.25
    ],
    "event.module": [
      "logstash"
    ],
    "logstash.node.stats.logstash.name": [
      "logstash-0"
    ],
    "logstash_stats.os.cpu.load_average.15m": [
      5.9609375
    ],
    "logstash.node.stats.jvm.mem.heap_used_in_bytes": [
      2038912656
    ],
    "logstash.node.stats.pipelines": [
      {
        "events.duration_in_millis": [
          283
        ],
        "reloads.failures": [
          0
        ],
        "vertices": [
          {
            "id": [
              "kafka-nonprod"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "9a190692-8c9d-4d5b-b2df-f760f83e359a"
            ],
            "queue_push_duration_in_millis": [
              0
            ]
          },
          {
            "duration_in_millis": [
              81
            ],
            "id": [
              "173a762c4b4a8401cb50e7091c6372eb665d02155a5d80cd99c94cf34fe0fa22"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "9a190692-8c9d-4d5b-b2df-f760f83e359a"
            ],
            "events_in": [
              0
            ]
          },
          {
            "duration_in_millis": [
              11
            ],
            "id": [
              "106c904cd1afa52d0d97fbbcad7cd087950303bad607fc67ad628cf91189595c"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "9a190692-8c9d-4d5b-b2df-f760f83e359a"
            ],
            "events_in": [
              0
            ]
          },
          {
            "events_out": [
              0
            ],
            "cluster_uuid": [
              "MkUomKBLT6aa2MwBprBc_A"
            ],
            "pipeline_ephemeral_id": [
              "9a190692-8c9d-4d5b-b2df-f760f83e359a"
            ],
            "events_in": [
              0
            ],
            "duration_in_millis": [
              98
            ],
            "id": [
              "elastic-nonprod"
            ]
          }
        ],
        "queue.events_count": [
          0
        ],
        "queue.queue_size_in_bytes": [
          0
        ],
        "queue.max_queue_size_in_bytes": [
          0
        ],
        "events.filtered": [
          0
        ],
        "queue.type": [
          "memory"
        ],
        "events.queue_push_duration_in_millis": [
          0
        ],
        "events.in": [
          0
        ],
        "reloads.successes": [
          0
        ],
        "id": [
          "nonprodlogs"
        ],
        "ephemeral_id": [
          "9a190692-8c9d-4d5b-b2df-f760f83e359a"
        ],
        "hash": [
          "2d28d16bdcbfc5043f824ef368c72c7add4206d00959628775d9e8a7da3c162f"
        ],
        "events.out": [
          0
        ]
      },
      {
        "events.duration_in_millis": [
          286
        ],
        "reloads.failures": [
          0
        ],
        "vertices": [
          {
            "id": [
              "kafka-perf-nonprod"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
            ],
            "queue_push_duration_in_millis": [
              0
            ]
          },
          {
            "duration_in_millis": [
              68
            ],
            "id": [
              "5cd697a78e4614ce8bad85163662df4f4b544b4f5b26f1f6b91b60aa4390171b"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
            ],
            "events_in": [
              0
            ]
          },
          {
            "duration_in_millis": [
              11
            ],
            "id": [
              "17ffbaaf92f5cdc150b92f22896e720ad3dc533d904d53293634d4034547d52f"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
            ],
            "events_in": [
              0
            ]
          },
          {
            "events_out": [
              0
            ],
            "cluster_uuid": [
              "MkUomKBLT6aa2MwBprBc_A"
            ],
            "pipeline_ephemeral_id": [
              "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
            ],
            "events_in": [
              0
            ],
            "duration_in_millis": [
              106
            ],
            "id": [
              "elastic-perf-nonprod"
            ]
          }
        ],
        "queue.events_count": [
          0
        ],
        "queue.queue_size_in_bytes": [
          0
        ],
        "queue.max_queue_size_in_bytes": [
          0
        ],
        "events.filtered": [
          0
        ],
        "queue.type": [
          "memory"
        ],
        "events.queue_push_duration_in_millis": [
          0
        ],
        "events.in": [
          0
        ],
        "reloads.successes": [
          0
        ],
        "id": [
          "perfcluster"
        ],
        "ephemeral_id": [
          "c01dfe30-1ff5-497e-bd12-dbc8fd555799"
        ],
        "hash": [
          "b81067844b9bd7ce93a455f6a867314d0eafd4bf8a265f10fbbd6b49c842ee59"
        ],
        "events.out": [
          0
        ]
      },
      {
        "events.duration_in_millis": [
          174
        ],
        "reloads.failures": [
          0
        ],
        "vertices": [
          {
            "id": [
              "kafka-signet-perf-nonprod"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "aab1a03c-3984-4517-b30d-3539ba3c12b0"
            ],
            "queue_push_duration_in_millis": [
              0
            ]
          },
          {
            "duration_in_millis": [
              36
            ],
            "id": [
              "3f0db88085ac75ecc9e76656a6b6e2e8f5396e391d933339b4cbf0b1499d5e43"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "aab1a03c-3984-4517-b30d-3539ba3c12b0"
            ],
            "events_in": [
              0
            ]
          },
          {
            "duration_in_millis": [
              12
            ],
            "id": [
              "3ae09666f2cf5668681d20b0957e99106f7eab583678d1894594291737bef1ac"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "aab1a03c-3984-4517-b30d-3539ba3c12b0"
            ],
            "events_in": [
              0
            ]
          },
          {
            "events_out": [
              0
            ],
            "cluster_uuid": [
              "MkUomKBLT6aa2MwBprBc_A"
            ],
            "pipeline_ephemeral_id": [
              "aab1a03c-3984-4517-b30d-3539ba3c12b0"
            ],
            "events_in": [
              0
            ],
            "duration_in_millis": [
              58
            ],
            "id": [
              "elastic-signet-perf-nonprod"
            ]
          }
        ],
        "queue.events_count": [
          0
        ],
        "queue.queue_size_in_bytes": [
          0
        ],
        "queue.max_queue_size_in_bytes": [
          0
        ],
        "events.filtered": [
          0
        ],
        "queue.type": [
          "memory"
        ],
        "events.queue_push_duration_in_millis": [
          0
        ],
        "events.in": [
          0
        ],
        "reloads.successes": [
          0
        ],
        "id": [
          "signetperf"
        ],
        "ephemeral_id": [
          "aab1a03c-3984-4517-b30d-3539ba3c12b0"
        ],
        "hash": [
          "a8796c1de82a8534fca361a09b7f93ac23b690570ad9cf8faedff718d22fc420"
        ],
        "events.out": [
          0
        ]
      },
      {
        "events.duration_in_millis": [
          264
        ],
        "reloads.failures": [
          0
        ],
        "vertices": [
          {
            "id": [
              "kafka-filebeat-metrics-nonprod"
            ],
            "events_out": [
              0
            ],
            "pipeline_ephemeral_id": [
              "a53d534e-e568-492d-a635-a065093a1be4"
            ],
            "queue_push_duration_in_millis": [
              0
            ]
          },
          {
            "events_out": [
              0
            ],
            "cluster_uuid": [
              "MkUomKBLT6aa2MwBprBc_A"
            ],
            "pipeline_ephemeral_id": [
              "a53d534e-e568-492d-a635-a065093a1be4"
            ],
            "events_in": [
              0
            ],
            "duration_in_millis": [
              141
            ],
            "id": [
              "5458cde1d3492f585a53f8348912826bb43c5ec17de43d1b037c28feb995ee17"
            ]
          },
          {
            "events_out": [
              0
            ],
            "cluster_uuid": [
              "MkUomKBLT6aa2MwBprBc_A"
            ],
            "pipeline_ephemeral_id": [
              "a53d534e-e568-492d-a635-a065093a1be4"
            ],
            "events_in": [
              0
            ],
            "duration_in_millis": [
              23
            ],
            "id": [
              "ee71743edcff7959c96e78ca85788300924cb785f702453653e9ab60e23987fc"
            ]
          }
        ],
        "queue.events_count": [
          0
        ],
        "queue.queue_size_in_bytes": [
          0
        ],
        "queue.max_queue_size_in_bytes": [
          0
        ],
        "events.filtered": [
          0
        ],
        "queue.type": [
          "memory"
        ],
        "events.queue_push_duration_in_millis": [
          0
        ],
        "events.in": [
          0
        ],
        "reloads.successes": [
          0
        ],
        "id": [
          "filebeatmetrics"
        ],
        "ephemeral_id": [
          "a53d534e-e568-492d-a635-a065093a1be4"
        ],
        "hash": [
          "da032c952cfa48c81d95b7aefc4508453a3c8c593e05fa0619d901238bda6fae"
        ],
        "events.out": [
          0
        ]
      }
    ],
    "timestamp": [
      "2023-09-19T09:32:13.970Z"
    ],
    "logstash_stats.logstash.version": [
      "8.7.1"
    ],
    "logstash.node.stats.os.cpu.load_average.15m": [
      5.9609375
    ],
    "service.name": [
      "logstash"
    ],
    "logstash.node.stats.jvm.gc.collectors.old.collection_count": [
      0
    ],
    "logstash.node.stats.jvm.gc.collectors.old.collection_time_in_millis": [
      0
    ],
    "logstash.node.stats.jvm.gc.collectors.young.collection_count": [
      220
    ],
    "metricset.period": [
      10000
    ],
    "logstash_stats.os.cgroup.cpu.cfs_quota_micros": [
      0
    ],
    "logstash.node.stats.logstash.host": [
      "logstash-0"
    ],
    "logstash.node.stats.jvm.uptime_in_millis": [
      8642993
    ],
    "logstash.node.stats.events.in": [
      0
    ],
    "logstash.node.stats.logstash.uuid": [
      "d7d77c8b-9b20-4c7c-a454-d6516bc6c886"
    ],
    "metricset.name": [
      "node_stats"
    ],
    "event.duration": [
      58322714
    ],
    "logstash.node.stats.os.cpu.load_average.1m": [
      5.25
    ],
    "@timestamp": [
      "2023-09-19T09:32:13.970Z"
    ],
    "logstash.node.stats.os.cgroup.cpu.cfs_quota_micros": [
      0
    ],
    "service.version": [
      "8.7.1"
    ],
    "logstash.node.stats.os.cpu.load_average.5m": [
      6.6601562
    ],
    "agent.ephemeral_id": [
      "d5918dc5-cb84-4d12-aca3-a086041d8b56"
    ],
    "logstash_stats.jvm.uptime_in_millis": [
      8642993
    ],
    "event.dataset": [
      "logstash.node.stats"
    ],
    "logstash.elasticsearch.cluster.id": [
      "nonprod-logs-cluster"
    ],
    "cloud.instance.name": [
      "aks-azpool3-35535046-vmss_4"
    ]
  }
}
4

Did you specify this as the cluster ID in the logstash.yml?
Is there a matching Elasticsearch cluster for this?

I think the Stack Monitoring UI won't know where to display that data otherwise.
What about the result from the Health API?

5

Thanks @miltonhultgren that gave me an idea, actually had set monitoring.cluster_uuid: "nonprod-logs-cluster" but then realized the actual UUID was different a randomly generated id.

6

For Beats the approach we follow is push the output to kafka -> logstash and then read it into ES. The datastream that get's created is not of the format .monitoring-beats-8-mb. But of the format metricbeat-*. Will the Stack Monitoring still be able to display the data?

7

Right, so for Logstash, please align the ID with the randomly generated ID of the Elasticsearch cluster.

For the Beats data, it has to end up in .monitoring-* for the Stack Monitoring UI to use it, so you'll need to figure out how to route the data to the right data stream. I'm not sure how that is done, what does your Logstash output look like today?

8

Got it thanks again, for beats we have just routed the logs directly to ES for now and things seem fine.

9

Hello @gsekar ,

Could you please help with your config file through which you are able to monitor beats and logstash.
I need to implement a monitoring dashboard displaying all beats- metrcicbeat,filebeat and logstash with there hostname and status if up or down in kibana table visual. I am new to this stack and confused about XPACK settings ,where and what settings need to be done.
Below table is requirement and also would like to monitor the same on kibana UI if possible.

image

The idea is to show for example out of 10 filebeat on respective host if 3 goes down, then need to show the hostname along with its status up or down, same applies for rest beats and logstash.The documentation is bit confusing to implement for me.

Your help would be highly appreciated.

Many Thanx.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.