Metricbeat host not showing on kibana as well as not creating index

[jonoyo]

After install the metricbeat on localhost elk server its showing on kibana dashboard but when metricbeat installing on another machine metricbeat host are not showing except elk localhost metricbeat host.

metricbeat version: 7.16.2
Kibana version: v 7.16.1
Elastic version: 7.16.1

[stephenb (Stephen Brown)]

@jonoyo

Most likely you have not changed the network settings and therefore Elasticsearch is only bound to localhost and cannot be reached from other hosts

Please see here

Important Elasticsearch configuration | Elasticsearch Guide [7.16] | Elastic

[warkolm (Mark Walkom)]

If that's not the case, you will need to provide your Metricbeat config and logs.

[jonoyo]

I have already configured network host to any host (0.0.0.0). Also configured server host to any host (0.0.0.0). Here is the configuration I have done..

Elasticsearch.yml###

root@elkserver:~# cat /etc/Elasticsearch/Elasticsearch.yml | grep -v "#"
cluster.name: elk-server
node.name: elk-node
path.data: /var/lib/Elasticsearch
path.logs: /var/log/Elasticsearch
network.host: 0.0.0.0
discovery.type: single-node

xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.monitoring.Elasticsearch.collection.enabled: true

####kibana.yml#####
server.port: 5601
server.host: "0.0.0.0"
server.publicBaseUrl: "http://172.16.10.10:5601/"
server.name: "kibana-server"
Elasticsearch.hosts: ["http://localhost:9200"]
Elasticsearch.username: "kibana"
Elasticsearch.password: "zeus"

######metricbeat.yml of remote host#####

metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
name: nagios
tags: ["nagios"]
setup.kibana:
host: "172.16.10.10:5601"
output.Elasticsearch:
hosts: ["172.16.10.10:9200"]
protocol: "http"
username: "elastic"
password: "atom"
processors:

• add_host_metadata: ~
• add_cloud_metadata: ~
• add_docker_metadata: ~
• add_kubernetes_metadata: ~
  logging.level: debug
  logging.selectors: ["*"]
  logging.to_files: true
  logging.files:
  path: /var/log/metricbeat
  name: metricbeat
  keepfiles: 7
  permissions: 0644
  monitoring.enabled: true
  monitoring.Elasticsearch:

metricbeat log of remote host

2021-12-22T10:51:11.362+0600 DEBUG [Elasticsearch] Elasticsearch/client.go:232 PublishEvents: 50 events have been published to Elasticsearch in 41.955739ms.
2021-12-22T10:51:11.364+0600 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [24371: 0, 50]
2021-12-22T10:51:11.364+0600 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:50
2021-12-22T10:51:11.364+0600 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2021-12-22T10:51:11.380+0600 DEBUG [Elasticsearch] Elasticsearch/client.go:232 PublishEvents: 50 events have been published to Elasticsearch in 17.998645ms.
2021-12-22T10:51:11.381+0600 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [24372: 0, 50]
2021-12-22T10:51:11.381+0600 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:50
2021-12-22T10:51:11.381+0600 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2021-12-22T10:51:11.399+0600 DEBUG [Elasticsearch] Elasticsearch/client.go:232 PublishEvents: 50 events have been published to Elasticsearch in 18.60274ms.
2021-12-22T10:51:11.399+0600 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [24373: 0, 50]
2021-12-22T10:51:11.400+0600 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:50
2021-12-22T10:51:11.400+0600 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2021-12-22T10:51:11.407+0600 DEBUG [Elasticsearch] Elasticsearch/client.go:232 PublishEvents: 18 events have been published to Elasticsearch in 7.561235ms.
2021-12-22T10:51:11.407+0600 DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [24374: 0, 18]
2021-12-22T10:51:11.407+0600 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:18
2021-12-22T10:51:11.407+0600 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack

[stephenb (Stephen Brown)]

Which module are you using?

Did you edit the system.yml on the remote host?

Can you show where / how the other servers are not showing?

If you go to Discover index pattern metricbeat-* and click on host.name field do you see the different hosts listed? Or in the KQL bar if you type in host.name : do you not see the other hosts?

The logs from the remote host look like they are sending to Elasticsearch but at the beginning of those logs you should see where metricbeat makes a connection to Elasticsearch you would want to verify that.

[jonoyo]

Thanks for your reply. Yes I edited system.yml and beat-xpack.yml on the remote host. In the index pattern I can see the host but when I check at the inventory I don't see the remote host. I only see the localhost. Here is the screenshot.

But when I check at the beats remote host metricbeat is active. Here is the screenshot.

When I try to check the remote host dashboard I also can't find the dashboard rather than elasticserch server host.

[stephenb (Stephen Brown)]

@jonoyo

system.yml and beat-xpack.yml

What Changes did you make? What is beat-xpack.yml? I am not familiar with that...

I suspect the setup or mapping is not correct...

If you just set up the remote host metricbeat and do not edit anything except set the elasticsearch and kibana endpoints in the metricbeat.yml does it work?

I know this sounds silly but the remote host has a different name right?

[jonoyo]

I have found on github beat-xpack.yml module monitor the beats on the system. When I disabled the modules that means just set up the host metricbeat only, the result is same. Here is the system.yml and beat-xpack.yml

####system.yml#####

Module: system

Docs: System module | Metricbeat Reference [7.16] | Elastic

• module: system
  period: 10s
  metricsets:
  • cpu
  • load
  • memory
  • network
  • process
  • process_summary
  • socket_summary
  • entropy
  • core
  • diskio
  • socket
  • service
  • users
    process.include_top_n:
    by_cpu: 5 # include top 5 processes by CPU
    by_memory: 5 # include top 5 processes by memory

Configure the mount point of the host’s filesystem for use in monitoring a host from within a container

#system.hostfs: "/hostfs"

• module: system
  period: 1m
  metricsets:

  • filesystem
  • fsstat
    processors:
  • drop_event.when.regexp:
    system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'

• module: system
  period: 15m
  metricsets:

  • uptime

#- module: system

period: 5m

metricsets:

- raid

raid.mount_point: '/'

#####beat-xpack.yml######

Module: beat

Docs: Beat module | Metricbeat Reference [7.16] | Elastic

• module: beat
  xpack.enabled: true
  period: 10s
  hosts: ["http://localhost:5066"]
  username: "beats_system"
  password: "twincities"

[stephenb (Stephen Brown)]

Can you please format your code? It is Hard to read.. I can not tell what it is ... use the </> button

If you are using a config you found elsewhere is is hard to help.

The beat module does not do system metrics...

Please just install metricbeat on a remote host ... NOT the host that Elasticsearch is running on...
make no changes, do not use the github config, and ONLY set the output.elasticsearch in the metribeat.yml and then run metricbeat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["elasticsearch_host:9200"] <!-- The Host Elasticsearch is running on

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "changeme"

You can also simply set this in the metricbeat.yml and metricbeat monitoring will be enabled and same to the same Elasticsearch.

# Set to true to enable the monitoring reporter.
monitoring.enabled: true

[jonoyo]

Thanks a lot. Now metricbeat is showing. Here is the screenshot.

My goal is to monitor my "ceph nodes" resource using metric ceph module of metricbeat. That is why I was trying first to monitor linux host using metricbeat. What could be best the solution if I want to monitor my ceph nodes?

[stephenb (Stephen Brown)]

Excellent

First stop metricbeat

Now I would go into the system.yml and turn on the other metrics that you might want.

Then I would enable the ceph module and go in and set the right setting.

Ceph module | Metricbeat Reference [7.16] | Elastic

I would make sure you also set this in the metricbeat.tml

setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "elasticsearchhost:5601"

then run setup , which will load the ceph module assets
metricbeat setup
then start metricbeat

[jonoyo]

Thanks a lot for your help. It is such a great community.

[jonoyo]

I am facing another problem now. I have already enabled system module of metricbeat in other hosts and uncomment the metricset. But there are many items are not showing in dashboard like below snapshot. Is there any other beat need to be configured? But all items are showing at elk localhost.

[stephenb (Stephen Brown)]

Please see this topic / post.. there is a bug.

Metrics Disappearing

Apologies.. Well I think I found it and I think it is a bug / regression... This is on my 7.16.0 Cluster. ...I have opened a bug here There is a rather lengthy explanation I will not bore you with but the fix is pretty simple although unfortunate. In short the data should be dropping the last "Partial Bucket" and it is not ... You can fix this by doing the following steps on each visualization it will take about 1 min or less each. In short there is a setting wrong in several of the Visualiz…