Metricbeat host not showing on kibana as well as not creating index

After install the metricbeat on localhost elk server its showing on kibana dashboard but when metricbeat installing on another machine metricbeat host are not showing except elk localhost metricbeat host.

metricbeat version: 7.16.2
Kibana version: v 7.16.1
Elastic version: 7.16.1

@jonoyo

Most likely you have not changed the network settings and therefore Elasticsearch is only bound to localhost and cannot be reached from other hosts

Please see here

If that's not the case, you will need to provide your Metricbeat config and logs.

I have already configured network host to any host (0.0.0.0). Also configured server host to any host (0.0.0.0). Here is the configuration I have done..

Elasticsearch.yml###

root@elkserver:~# cat /etc/Elasticsearch/Elasticsearch.yml | grep -v "#"
cluster.name: elk-server
node.name: elk-node
path.data: /var/lib/Elasticsearch
path.logs: /var/log/Elasticsearch
network.host: 0.0.0.0
discovery.type: single-node

xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.monitoring.Elasticsearch.collection.enabled: true

####kibana.yml#####
server.port: 5601
server.host: "0.0.0.0"
server.publicBaseUrl: "http://172.16.10.10:5601/"
server.name: "kibana-server"
Elasticsearch.hosts: ["http://localhost:9200"]
Elasticsearch.username: "kibana"
Elasticsearch.password: "zeus"

######metricbeat.yml of remote host#####

metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
name: nagios
tags: ["nagios"]
setup.kibana:
host: "172.16.10.10:5601"
output.Elasticsearch:
hosts: ["172.16.10.10:9200"]
protocol: "http"
username: "elastic"
password: "atom"
processors:

  • add_host_metadata: ~
  • add_cloud_metadata: ~
  • add_docker_metadata: ~
  • add_kubernetes_metadata: ~
    logging.level: debug
    logging.selectors: ["*"]
    logging.to_files: true
    logging.files:
    path: /var/log/metricbeat
    name: metricbeat
    keepfiles: 7
    permissions: 0644
    monitoring.enabled: true
    monitoring.Elasticsearch:
metricbeat log of remote host

2021-12-22T10:51:11.362+0600 DEBUG [Elasticsearch] Elasticsearch/client.go:232 PublishEvents: 50 events have been published to Elasticsearch in 41.955739ms.
2021-12-22T10:51:11.364+0600 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [24371: 0, 50]
2021-12-22T10:51:11.364+0600 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:50
2021-12-22T10:51:11.364+0600 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2021-12-22T10:51:11.380+0600 DEBUG [Elasticsearch] Elasticsearch/client.go:232 PublishEvents: 50 events have been published to Elasticsearch in 17.998645ms.
2021-12-22T10:51:11.381+0600 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [24372: 0, 50]
2021-12-22T10:51:11.381+0600 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:50
2021-12-22T10:51:11.381+0600 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2021-12-22T10:51:11.399+0600 DEBUG [Elasticsearch] Elasticsearch/client.go:232 PublishEvents: 50 events have been published to Elasticsearch in 18.60274ms.
2021-12-22T10:51:11.399+0600 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [24373: 0, 50]
2021-12-22T10:51:11.400+0600 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:50
2021-12-22T10:51:11.400+0600 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2021-12-22T10:51:11.407+0600 DEBUG [Elasticsearch] Elasticsearch/client.go:232 PublishEvents: 18 events have been published to Elasticsearch in 7.561235ms.
2021-12-22T10:51:11.407+0600 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [24374: 0, 18]
2021-12-22T10:51:11.407+0600 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:18
2021-12-22T10:51:11.407+0600 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack

Which module are you using?

Did you edit the system.yml on the remote host?

Can you show where / how the other servers are not showing?

If you go to Discover index pattern metricbeat-* and click on host.name field do you see the different hosts listed? Or in the KQL bar if you type in host.name : do you not see the other hosts?

The logs from the remote host look like they are sending to Elasticsearch but at the beginning of those logs you should see where metricbeat makes a connection to Elasticsearch you would want to verify that.

Thanks for your reply. Yes I edited system.yml and beat-xpack.yml on the remote host. In the index pattern I can see the host but when I check at the inventory I don't see the remote host. I only see the localhost. Here is the screenshot.

But when I check at the beats remote host metricbeat is active. Here is the screenshot.

When I try to check the remote host dashboard I also can't find the dashboard rather than elasticserch server host.

What Changes did you make? What is beat-xpack.yml? I am not familiar with that...

I suspect the setup or mapping is not correct...

If you just set up the remote host metricbeat and do not edit anything except set the Elasticsearch and kibana endpoints in the metricbeat.yml does it work?

I know this sounds silly but the remote host has a different name right?

I have found on github beat-xpack.yml module monitor the beats on the system. When I disabled the modules that means just set up the host metricbeat only, the result is same. Here is the system.yml and beat-xpack.yml

####system.yml#####

Module: system

Docs: System module | Metricbeat Reference [7.16] | Elastic

  • module: system
    period: 10s
    metricsets:
    • cpu
    • load
    • memory
    • network
    • process
    • process_summary
    • socket_summary
    • entropy
    • core
    • diskio
    • socket
    • service
    • users
      process.include_top_n:
      by_cpu: 5 # include top 5 processes by CPU
      by_memory: 5 # include top 5 processes by memory

Configure the mount point of the host’s filesystem for use in monitoring a host from within a container

#system.hostfs: "/hostfs"

  • module: system
    period: 1m
    metricsets:

    • filesystem
    • fsstat
      processors:
    • drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'
  • module: system
    period: 15m
    metricsets:

    • uptime

#- module: system

period: 5m

metricsets:

- raid

raid.mount_point: '/'

#####beat-xpack.yml######

Module: beat

Docs: Beat module | Metricbeat Reference [7.16] | Elastic

  • module: beat
    xpack.enabled: true
    period: 10s
    hosts: ["http://localhost:5066"]
    username: "beats_system"
    password: "twincities"

Can you please format your code? It is Hard to read.. I can not tell what it is ... use the </> button

If you are using a config you found elsewhere is is hard to help.

The beat module does not do system metrics...

Please just install metricbeat on a remote host ... NOT the host that Elasticsearch is running on...
make no changes, do not use the github config, and ONLY set the output.elasticsearch in the metribeat.yml and then run metricbeat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elasticsearch_host:9200"] <!-- The Host Elasticsearch is running on

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "changeme"

You can also simply set this in the metricbeat.yml and metricbeat monitoring will be enabled and same to the same Elasticsearch.

# Set to true to enable the monitoring reporter.
monitoring.enabled: true
1 Like

Thanks a lot. Now metricbeat is showing. Here is the screenshot.

My goal is to monitor my "ceph nodes" resource using metric ceph module of metricbeat. That is why I was trying first to monitor linux host using metricbeat. What could be best the solution if I want to monitor my ceph nodes?

Excellent

First stop metricbeat

Now I would go into the system.yml and turn on the other metrics that you might want.

Then I would enable the ceph module and go in and set the right setting.

I would make sure you also set this in the metricbeat.tml

setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "elasticsearchhost:5601"

then run setup , which will load the ceph module assets
metricbeat setup
then start metricbeat

Thanks a lot for your help. It is such a great community. :slight_smile:

1 Like

I am facing another problem now. I have already enabled system module of metricbeat in other hosts and uncomment the metricset. But there are many items are not showing in dashboard like below snapshot. Is there any other beat need to be configured? But all items are showing at elk localhost.

Please see this topic / post.. there is a bug.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.