Beats instance detected Click 'Set up monitoring' below to start monitoring this instance

Elastic Stack Beats
,
1

All services are running. I log into 5601, click the drop down hamburger | Management | Stack Monitoring

beatst_2022-12-26_14-37-10
beatst_2022-12-26_14-37-10514×674 29 KB

But there is no 'Set up monitoring' to click on.

root@ub2204elk:/etc# cat metricbeat/metricbeat.yml | grep -v "#" |uniq

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.kibana:

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "metricbeat_internal"
  password: "his^^password" 
  protocol: "http"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

root@ub2204elk:/etc# cat heartbeat/heartbeat.yml | grep -v "#" |uniq

heartbeat.config.monitors:
  path: ${path.config}/monitors.d/*.yml
  reload.enabled: true
  reload.period: 5s

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

tags: ["elk"]

setup.kibana:

  host: "192.168.1.139:5601"

output.elasticsearch:
  hosts: ["localhost:9200"]

  protocol: "http"

  username: "elastic"
  password: "testpw"

processors:
  - add_observer_metadata:

logging.level: debug

logging.selectors: ["*"]

logging.to_files: true
logging.files:
  path: /var/log/heatbeat
  name: heartbeat
  keepfiles: 7
  permissions: 0644

http.enabled: true
http.port: 5069

root@ub2204elk:/etc# cat heartbeat/monitors.d/icmp.yml | grep -v "#" |uniq

  id: icmp-status

  name: icmp-status check

  enabled: true

  hosts: ["192.168.1.139", "192.168.1.96", "8.8.8.8"]

  ipv4: true
  mode: any

  timeout: 16s

  wait: 1s

  tags: ["icmp-status"]

root@ub2204elk:/etc# cat heartbeat/monitors.d/http.yml | grep -v "#" |uniq

  id: web-status

  name: web-status check

  enabled: true

  hosts: ["http://192.168.1.139:9200", "http://192.168.1.139:5601", "http://google.com"]

  ipv4: true

  #username: 'elastic'
  #password: 'testpw'

  tags: ["web-status"]

cat /etc/metricbeat/modules.d/beat-xpack.yml

# Module: beat
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/8.5/metricbeat-module-beat.html

- module: beat
  xpack.enabled: true
  period: 10s
  hosts: ["http://localhost:5066, http://localhost:5067, http://localhost:5068, http://localhost:5069"]
    #heartbeat listens on 5069?
  username: "beat_system"
  password: "beat_systempassword"

all elk stack is on one server including the problem heartbeat. This server's IP is 192.168.1.139.
Are there any other yml file to see?

2

Hi @ant2ne,

Thanks for reaching out!

What version of Kibana are you using?

Could you check if these docs help you? Monitor a cluster | Elasticsearch Guide [8.5] | Elastic

Thanks,
Alberto

3

Thanks for the reply. I suppose I should have included the references I am using...
https://www.youtube.com/watch?v=qzgTOA9eJgk
https://www.elastic.co/guide/en/beats/metricbeat/8.5/metricbeat-installation-configuration.html
https://www.elastic.co/guide/en/beats/heartbeat/8.5/heartbeat-installation-configuration.html

/usr/share/kibana/bin/kibana --version
8.5.3

This is not a cluster. Just a stand alone elk server for log and uptime monitoring (for now). But still in a secluded test environment.

I commented username and password from heartbeat/monitors.d/http.yml. Changed localhost to the IP of the interface. No effect.

(included /etc/metricbeat/modules.d/beat-xpack.yml above in case it needs referenced)

4

This seems worth mentioning...

beatst_2022-12-27_10-09-19
beatst_2022-12-27_10-09-19788×733 57.4 KB

I am the only google hit!! The only person in google world who is researching this message.

6

Can you run metricbeat modules list to see which modules are actually enabled? You should see the beat module be enabled.

You can also try configuring the modules to enable in the metricbeat.yml file directly, as explained here Configure modules | Metricbeat Reference [8.5] | Elastic

7 
metricbeat modules list
Enabled:
beat-xpack
elasticsearch-xpack
kibana-xpack

Disabled:
8

metricbeat modules enable beat

The error message has changed. I don't know if that is progress. I am still getting the original error, but the logs now show...

tail -f /var/log/heartbeat/heartbeat-20221227-18.ndjson /var/log/metricbeat/metricbeat-20221227-4.ndjson syslog | grep -i error

Dec 28 15:01:28 ub2204elk metricbeat[24139]: {"log.level":"error","@timestamp":"2022-12-28T15:01:28.708Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://192.168.1.139:5069/stats\": dial tcp 192.168.1.139:5069: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:01:28 ub2204elk metricbeat[24139]: {"log.level":"error","@timestamp":"2022-12-28T15:01:28.709Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://192.168.1.139:5069/state\": dial tcp 192.168.1.139:5069: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}

"connection refused" usually means firewall, or username/password. These ports are defined in beat.yml and beat-xpack.tml (/etc/metricbeat/modules.d)

ufw is disabled for testing ans something is listening on 5069

root@ub2204elk:/etc/metricbeat/modules.d# netstat -na | grep -e 5069
tcp        0      0 127.0.0.1:5069          0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:5069          127.0.0.1:60692         ESTABLISHED
tcp        0      0 127.0.0.1:60692         127.0.0.1:5069          ESTABLISHED
tcp        0      0 127.0.0.1:5069          127.0.0.1:60708         ESTABLISHED
tcp        0      0 127.0.0.1:60708         127.0.0.1:5069          ESTABLISHED
root@ub2204elk:/etc/metricbeat/modules.d# netstat -tuplen | grep 5069
tcp        0      0 127.0.0.1:5069          0.0.0.0:*               LISTEN      0          20638      628/heartbeat       

cat beat-xpack.yml
- module: beat
  xpack.enabled: true
  period: 10s
  hosts: ["http://192.168.1.139:5069"] #heartbeat listens on 5069?
  service.name: metricbeat
  username: "beat_system"
  password: "^^password"
    #username: "remote_monitoring_user"
    #password: "^^password"

and

cat beat.yml
- module: beat
  metricsets:
    - stats
    - state
  period: 10s
  hosts: ["http://192.168.1.139:5069"]
  username: "beat_system"
  password: "^^password"

Should I open a new thread with this new error in the logs?

below is the un-grepped logs

Dec 28 15:12:21 ub2204elk systemd[1]: Started Metricbeat is a lightweight shipper for metrics..
Dec 28 15:12:22 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:22.063Z","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:22 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:22.064Z","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: 1bf69668-e0d8-44a6-b7ba-a367645599c1","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"warn","@timestamp":"2022-12-28T15:12:25.065Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.066Z","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":124},"message":"Syscall filter successfully installed","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.066Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1082},"message":"Beat info","service.name":"metricbeat","system_info":{"beat":{"path":{"config":"/etc/metricbeat","data":"/var/lib/metricbeat","home":"/usr/share/metricbeat","logs":"/var/log/metricbeat"},"type":"metricbeat","uuid":"1bf69668-e0d8-44a6-b7ba-a367645599c1"},"ecs.version":"1.6.0"}}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.066Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1091},"message":"Build info","service.name":"metricbeat","system_info":{"build":{"commit":"6d03209df870c63ef9d59d609268c11dfdc835dd","libbeat":"8.5.3","time":"2022-12-04T04:55:28.000Z","version":"8.5.3"},"ecs.version":"1.6.0"}}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.066Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1094},"message":"Go runtime info","service.name":"metricbeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":1,"version":"go1.18.7"},"ecs.version":"1.6.0"}}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.067Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1098},"message":"Host info","service.name":"metricbeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-12-27T17:25:58Z","containerized":false,"name":"ub2204elk","ip":["127.0.0.1/8","::1/128","192.168.1.139/24","2600:6c40:0:7de:a00:27ff:fea4:3a3b/64","fe80::a00:27ff:fea4:3a3b/64"],"kernel_version":"5.15.0-56-generic","mac":["08:00:27:a4:3a:3b"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"22.04.1 LTS (Jammy Jellyfish)","major":22,"minor":4,"patch":1,"codename":"jammy"},"timezone":"UTC","timezone_offset_sec":0,"id":"40d85caac60a47808a0f44d99d777136"},"ecs.version":"1.6.0"}}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.067Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1127},"message":"Process info","service.name":"metricbeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null},"cwd":"/","exe":"/usr/share/metricbeat/bin/metricbeat","name":"metricbeat","pid":24349,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2022-12-28T15:12:21.830Z"},"ecs.version":"1.6.0"}}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.067Z","log.origin":{"file.name":"instance/beat.go","file.line":294},"message":"Setup Beat: metricbeat; Version: 8.5.3","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.069Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: http://localhost:9200","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.070Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: ub2204elk","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.087Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":144},"message":"Starting metrics logging every 30s","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.087Z","log.origin":{"file.name":"instance/beat.go","file.line":471},"message":"metricbeat start running.","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.107Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},"message":"Config reloader started","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"info","@timestamp":"2022-12-28T15:12:25.184Z","log.origin":{"file.name":"cfgfile/reload.go","file.line":224},"message":"Loading of config files completed.","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"error","@timestamp":"2022-12-28T15:12:25.199Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://192.168.1.139:5069/state\": dial tcp 192.168.1.139:5069: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"error","@timestamp":"2022-12-28T15:12:25.200Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://192.168.1.139:5069/stats\": dial tcp 192.168.1.139:5069: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"error","@timestamp":"2022-12-28T15:12:25.200Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset beat.stats: error making http request: Get \"http://192.168.1.139:5069/stats\": dial tcp 192.168.1.139:5069: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
Dec 28 15:12:25 ub2204elk metricbeat[24349]: {"log.level":"error","@timestamp":"2022-12-28T15:12:25.200Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://192.168.1.139:5069/state\": dial tcp 192.168.1.139:5069: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
9

progress, the above errors have gone away. Including the top "Set up monitoring" by changing metricbeat/modules.d/beat-xpack.yml and metricbeat/modules.d/beat.yml from the direct interface ip to 127.0.0.1. It didn't like 'localhost' or '192.168.1.X' IP in these config yml files. I am not sure if this is a bug, but it is a frustration. My current issue is..

beats.works.maybe_2022-12-28_10-15-28
beats.works.maybe_2022-12-28_10-15-281689×749 77 KB

It isn't showing failed beats, my tcp.yml monitor says 8.8.8.8:22 (which should not be allowed) but it does not show up as a failure in kibana. This issue is different enough that I will open a new thread. I'm going to label this thread as solved.

Wish me luck.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.