Metricbeat failed to start at system reboot

[ijazadm (Ijaz Ahmad Khan)]

Hi ,

Metricbeat is running fine , and is enabled to start at system boot through systemd, rhel7.

but when I reboot the system , it fails to start and I have to restart is manually.

Any ideas?

log:

[root@wp-p3s-2c ~]# systemctl status metricbeat
● metricbeat.service - metricbeat
Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Fri 2018-01-19 14:35:30 GMT; 1min 15s ago
Docs: https://www.elastic.co/guide/en/beats/metricbeat/current/index.html
Process: 2270 ExecStart=/usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.data /var/lib/metricbeat -path.logs /var/log/metricbeat (code=exited, status=1/FAILURE)
Main PID: 2270 (code=exited, status=1/FAILURE)

Jan 19 14:35:30 wp-p3s-2c systemd[1]: Unit metricbeat.service entered failed state.
Jan 19 14:35:30 wp-p3s-2c systemd[1]: metricbeat.service failed.
Jan 19 14:35:30 wp-p3s-2c systemd[1]: metricbeat.service holdoff time over, scheduling restart.
Jan 19 14:35:30 wp-p3s-2c systemd[1]: start request repeated too quickly for metricbeat.service
Jan 19 14:35:30 wp-p3s-2c systemd[1]: Failed to start metricbeat.
Jan 19 14:35:30 wp-p3s-2c systemd[1]: Unit metricbeat.service entered failed state.
Jan 19 14:35:30 wp-p3s-2c systemd[1]: metricbeat.service failed.
[root@wp-p3s-2c ~]#
[root@wp-p3s-2c ~]#
[root@wp-p3s-2c ~]# systemctl restart metricbeat
[root@wp-p3s-2c ~]#

[root@wp-p3s-2c ~]# systemctl status metricbeat
● metricbeat.service - metricbeat
Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2018-01-19 14:36:54 GMT; 1s ago
Docs: https://www.elastic.co/guide/en/beats/metricbeat/current/index.html
Main PID: 2545 (metricbeat)
Memory: 11.0M
CGroup: /system.slice/metricbeat.service
└─2545 /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/metricbeat -path.con...

Jan 19 14:36:54 wp-p3s-2c systemd[1]: Started metricbeat.
Jan 19 14:36:54 wp-p3s-2c systemd[1]: Starting metricbeat...
[root@wp-p3s-2c ~]#

[steffens (Steffen Siering)]

Can you share metricbeat configuration and metricbeat logs. Just having the systemd logs gives you no details why it actually failed to startup.

Please format logs, configs and terminal input/output using the </>-Button or markdown code fences. This forum uses Markdown to format posts. Without proper formatting, it can be very hard to read your posts.

[ijazadm (Ijaz Ahmad Khan)]

here is the metricbeat configuration

metricbeat.autodiscover:
  providers:
- type: docker

metricbeat.modules:
- module: system
  metricsets:
- cpu
- filesystem
- memory
- network
- process
  enabled: true
  period: 10s
  processes: ['.*']
  cpu_ticks: false
- module: apache
  metricsets: ["status"]
  enabled: true
  period: 1s
  hosts: ["http://wp-p3s-1c.ebi.ac.uk"]
  server_status_path: "server-status"
- module: docker
  metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
- module: kibana
  metricsets: ["status"]
  period: 10s
  hosts: ["localhost:5601"]

output.elasticsearch:
  hosts: ["10.3.10.44:9200","10.3.10.28:9200"]

setup.kbana:
  host: "10.3.10.44:5601"

[ijazadm (Ijaz Ahmad Khan)]

there is no error in metricbeat logs.

[root@wp-p3s-1c ~]# 
[root@wp-p3s-1c ~]# 
[root@wp-p3s-1c ~]# grep -i error /var/log/metricbeat/metricbeat
[root@wp-p3s-1c ~]#

[ijazadm (Ijaz Ahmad Khan)]

the journalctl shows:

Jan 19 15:01:05 wp-p3s-1c.ebi.ac.uk metricbeat[1999]: Exiting: Error importing Kibana dashboards: fail to
Jan 19 15:01:05 wp-p3s-1c.ebi.ac.uk systemd[1]: metricbeat.service: main process exited, code=exited, sta
Jan 19 15:01:05 wp-p3s-1c.ebi.ac.uk systemd[1]: Unit metricbeat.service entered failed state.
Jan 19 15:01:05 wp-p3s-1c.ebi.ac.uk systemd[1]: metricbeat.service failed.
Jan 19 15:01:06 wp-p3s-1c.ebi.ac.uk systemd[1]: metricbeat.service holdoff time over, scheduling restart.
Jan 19 15:01:06 wp-p3s-1c.ebi.ac.uk systemd[1]: start request repeated too quickly for metricbeat.service
Jan 19 15:01:06 wp-p3s-1c.ebi.ac.uk systemd[1]: Failed to start metricbeat.
Jan 19 15:01:06 wp-p3s-1c.ebi.ac.uk systemd[1]: Unit metricbeat.service entered failed state.
Jan 19 15:01:06 wp-p3s-1c.ebi.ac.uk systemd[1]: metricbeat.service failed

i think it fails to import kibana dashboards , and hence fails to start? but why it then succeeds when i manually do:

systemctl restart metricbeat.

I just want it to start at system reboot without any manual steps.

[steffens (Steffen Siering)]

Check your startup order. Is metricbeat started before or after the network is up?

As you run metricbeat as a service, consider to disable automatic kibana/elasticsearch configuration. You can setup kibana manually (or automated via Ansible and co.) by running metricbeat setup ....