Metricbeat service is not starting after adding metricbeat keystore

Elastic Stack Beats
1

Hi Team,

I am following below link for metricbeat keystore setup on v7.14.0.

I created the keystore as,

 /usr/share/metricbeat/bin/metricbeat  keystore --path.config /etc/metricbeat create

Created the key

cat es_password |  /usr/share/metricbeat/bin/metricbeat  keystore --path.config /etc/metricbeat add ES_PWD --stdin

Able to list the key

[root@ip-10-10-10-242 metricbeat]# /usr/share/metricbeat/bin/metricbeat  keystore  list
ES_PWD
[root@ip-10-10-10-242 metricbeat]#

Mentioned in the metricbeat.yml as,

setup.dashboards.enabled: true
setup.kibana:
  host: "10.10.10.242"
  username: elastic
  password: "${ES_PWD}"

Able to see keystore file.

[root@ip-10-10-10-242 metricbeat]# ls -l /usr/share/metricbeat/bin/data/metricbeat.keystore

-rw-------. 1 root root 190 Sep 14 20:54 /usr/share/metricbeat/bin/data/metricbeat.keystore

[root@ip-10-10-10-242 metricbeat]#

After doing above metricbeat service is not getting started.

● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
   Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2021-09-14 21:01:52 UTC; 9min ago
     Docs: https://www.elastic.co/beats/metricbeat
  Process: 27738 ExecStart=/usr/share/metricbeat/bin/metricbeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
 Main PID: 27738 (code=exited, status=1/FAILURE)

Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: metricbeat.service: main process exited, code=exited, status=1/FAILURE
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: Unit metricbeat.service entered failed state.
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: metricbeat.service failed.
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: metricbeat.service holdoff time over, scheduling restart.
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: start request repeated too quickly for metricbeat.service
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: Failed to start Metricbeat is a lightweight shipper for metrics..
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: Unit metricbeat.service entered failed state.
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: metricbeat.service failed.

Can someone point out why this is happening?

Same problem is with filebeat also. However heartbeat is running fine after above configuration and heartbeat service restart.

Thanks,

2

Can you please check the actual Metricbeat logs, it should give you more of an indicator as to what's happening :slight_smile:

3

all available metricbeat logs files (/var/log/metricbeat/metricbeat, metricbeat.1, ....... metricbeat.4) are not getting updated even after metricbeat service restart. Last log is of 2021-08-24.

in journalctl -fu metricbeat can see this,

ERROR        instance/beat.go:989        Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')

Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')

We have defined password under kibana.yml (output given above)

complete logs -

Sep 15 05:20:44 ip-10-10-10-242 metricbeat[3279]: 2021-09-15T05:20:44.724Z        ERROR        instance/beat.go:989        Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')
Sep 15 05:20:44 ip-10-10-10-242 metricbeat[3279]: Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')
Sep 15 05:20:44 ip-10-10-10-242 systemd[1]: metricbeat.service: main process exited, code=exited, status=1/FAILURE
Sep 15 05:20:44 ip-10-10-10-242 systemd[1]: Unit metricbeat.service entered failed state.
Sep 15 05:20:44 ip-10-10-10-242[1]: metricbeat.service failed.
Sep 15 05:20:45 ip-10-10-10-242[1]: metricbeat.service holdoff time over, scheduling restart.
Sep 15 05:20:45 ip-10-10-10-242[1]: start request repeated too quickly for metricbeat.service
Sep 15 05:20:45 ip-10-10-10-242[1]: Failed to start Metricbeat is a lightweight shipper for metrics..
Sep 15 05:20:45 ip-10-10-10-242[1]: Unit metricbeat.service entered failed state.
Sep 15 05:20:45 ip-10-10-10-242[1]: metricbeat.service failed.

Thanks,

4

Ok no worries. Can you try starting Metricbeat directly?

5

directly? Sorry didnt get you.

I am starting it with systemctl command under linux. (systemctl restart metricbeat.service) and it is failing to start it.

Thanks,

6

Something like metricbeat -e I mean, calling the binary directly.

7

ok. Thank you.

Error -

2021-09-15T05:33:43.962Z	ERROR	instance/beat.go:989	Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/bin/kibana: No directory /usr/share/metricbeat/bin/kibana/7
Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/bin/kibana: No directory /usr/share/metricbeat/bin/kibana/7

complete logs -

[root@ip-10-10-10-242 metricbeat]# /usr/share/metricbeat/bin/metricbeat -e
2021-09-15T05:33:42.797Z	INFO	instance/beat.go:665	Home path: [/usr/share/metricbeat/bin] Config path: [/usr/share/metricbeat/bin] Data path: [/usr/share/metricbeat/bin/data] Logs path: [/usr/share/metricbeat/bin/logs]
2021-09-15T05:33:42.798Z	INFO	instance/beat.go:673	Beat ID: 5be2653a-2dd4-4663-b475-8ce47cd2b258
2021-09-15T05:33:42.799Z	INFO	[seccomp]	seccomp/seccomp.go:101	Syscall filter could not be installed because the kernel does not support seccomp
2021-09-15T05:33:42.799Z	INFO	[beat]	instance/beat.go:1014	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat/bin", "data": "/usr/share/metricbeat/bin/data", "home": "/usr/share/metricbeat/bin", "logs": "/usr/share/metricbeat/bin/logs"}, "type": "metricbeat", "uuid": "5be2653a-2dd4-4663-b475-8ce47cd2b258"}}}
2021-09-15T05:33:42.799Z	INFO	[beat]	instance/beat.go:1023	Build info	{"system_info": {"build": {"commit": "e127fc31fc6c00fdf8649808f9421d8f8c28b5db", "libbeat": "7.14.0", "time": "2021-07-29T21:09:05.000Z", "version": "7.14.0"}}}
2021-09-15T05:33:42.799Z	INFO	[beat]	instance/beat.go:1026	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.16.6"}}}
2021-09-15T05:33:42.800Z	INFO	[beat]	instance/beat.go:1030	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-02-15T11:12:32Z","containerized":false,"name":"ip-10-0-1-142.eu-west-3.compute.internal","ip":["127.0.0.1/8","::1/128","10.10.10.242/24","fe80::4b0:5ff:fedd:ac14/64"],"kernel_version":"3.10.0-693.5.2.el7.x86_64","mac":["06:b0:05:dd:ac:14"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":4,"patch":1708,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"f073c429a7456b53ec3e2c53460c5c8f"}}}
2021-09-15T05:33:42.801Z	INFO	[beat]	instance/beat.go:1059	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/etc/metricbeat", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 3574, "ppid": 3218, "seccomp": {"mode":"disabled"}, "start_time": "2021-09-15T05:33:41.650Z"}}}
2021-09-15T05:33:42.801Z	INFO	instance/beat.go:309	Setup Beat: metricbeat; Version: 7.14.0
2021-09-15T05:33:42.801Z	INFO	[publisher]	pipeline/module.go:113	Beat name: app_server
2021-09-15T05:33:42.802Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2021-09-15T05:33:42.802Z	INFO	kibana/client.go:122	Kibana url: http://10.10.10.242:5601
2021-09-15T05:33:42.804Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:105	add_cloud_metadata: hosting provider type detected as aws, metadata={"cloud":{"account":{"id":"641850179141"},"availability_zone":"eu-west-3a","image":{"id":"ami-0c60d771"},"instance":{"id":"i-03bbbf23fafc5bef9"},"machine":{"type":"t2.large"},"provider":"aws","region":"eu-west-3","service":{"name":"EC2"}}}
2021-09-15T05:33:43.337Z	INFO	kibana/client.go:122	Kibana url: http://10.10.10.242:5601
2021-09-15T05:33:43.962Z	INFO	[monitoring]	log/log.go:153	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":90,"time":{"ms":93}},"total":{"ticks":800,"time":{"ms":804},"value":800},"user":{"ticks":710,"time":{"ms":711}}},"handles":{"limit":{"hard":65535,"soft":65535},"open":10},"info":{"ephemeral_id":"cfe077f5-3ddf-42b8-9c35-d8c0baf6d7fb","uptime":{"ms":1244},"version":"7.14.0"},"memstats":{"gc_next":26976000,"memory_alloc":19665296,"memory_sys":75777032,"memory_total":118386160,"rss":137564160},"runtime":{"goroutines":20}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"system":{"cpu":{"cores":2},"load":{"1":0.09,"15":0.32,"5":0.23,"norm":{"1":0.045,"15":0.16,"5":0.115}}}}}}
2021-09-15T05:33:43.962Z	INFO	[monitoring]	log/log.go:154	Uptime: 1.246523834s
2021-09-15T05:33:43.962Z	INFO	[monitoring]	log/log.go:131	Stopping metrics logging.
2021-09-15T05:33:43.962Z	INFO	instance/beat.go:470	metricbeat stopped.
2021-09-15T05:33:43.962Z	ERROR	instance/beat.go:989	Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/bin/kibana: No directory /usr/share/metricbeat/bin/kibana/7
Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/bin/kibana: No directory /usr/share/metricbeat/bin/kibana/7
[root@ip-10-10-10-242 metricbeat]#
8

Thanks! It might be useful to share your Metricbeat config file, we might be able to find why it's complaining about Kibana dashboards there.

9

I believe the issue is that the keystore is owned by root, it needs to be owned by metricbeat which is the user metricbeat is started under via systemctl

sudo chown metricbeat:metricbeat /usr/share/metricbeat/bin/data/metricbeat.keystore

10

If i used the password instead of key in metricbeat.yml, and restart, service is up and running corrently.

11

Hi @stephenb,

There is no such user metricbeat.

[root@ip-10-10-10-242 metricbeat]# chown metricbeat:metricbeat /usr/share/metricbeat/bin/data/metricbeat.keystore 
chown: invalid user: 'metricbeat:metricbeat'

[root@ip-10-10-10-242 metricbeat]# id metricbeat
id: metricbeat: no such user
[root@ip-10-10-10-242 metricbeat]#
12

then
sudo chmod 644 /usr/share/metricbeat/bin/data/metricbeat.keystore

13

Hi @warkolm,

Here is the file

[root@ip-10-10-10-242 metricbeat]# cat /etc/metricbeat/metricbeat.yml
# Ansible managed

name: app_server
fields_under_root: true
fields:
  host_id: app_server

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

setup.dashboards.enabled: true
setup.kibana:
  host: "10.10.10.242"
  username: elastic
  password: xxx
# metricbeat setup.ilm

setup.ilm.enabled: false
setup.ilm.rollover_alias: "metricbeat"
setup.ilm.pattern: "{now/d}-000001" 

output.logstash:
  hosts: ['10.10.10.242:5044']
  loadbalance: False

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
[root@ip-10-10-10-242 metricbeat]#
14

Yeah sorry metricbeat install does not create a metricbeat user

If you want test run form command line the command is:

/usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /var/lib/metricbeat --path.logs /var/log/metricbeat

15

given 644

[root@ip-10-10-10-242 metricbeat]# ls -l  /usr/share/metricbeat/bin/data/metricbeat.keystore
-rw-r--r--. 1 root root 190 Sep 14 20:54 /usr/share/metricbeat/bin/data/metricbeat.keystore

Giving below error while listing keystore.

[root@ip-10-10-10-242 metricbeat]#  /usr/share/metricbeat/bin/metricbeat  keystore list
error initializing beat: could not initialize the keystore: file ("/usr/share/metricbeat/bin/data/metricbeat.keystore") can only be writable and readable by the owner but the permissions are "-rw-r--r--" (to fix the permissions use: 'chmod go-wrx /usr/share/metricbeat/bin/data/metricbeat.keystore')
[root@ip-10-10-10-242 metricbeat]#
16

@stephenb,

Sorry lost your comment about directory layout. Can't see it now.

I have installed metricbeat via rpm.

[root@ip-10-10-10-242 metricbeat]# rpm -qa |grep metricbeat
metricbeat-7.14.0-1.x86_64
[root@ip-10-10-10-242 metricbeat]#

its saying missing field accessing setup.kibana.password which we have set in config file.

[root@ip-10-10-10-242 metricbeat]#  /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /var/lib/metricbeat --path.logs /var/log/metricbeat
Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')
[root@ip-10-10-10-242 metricbeat]#
17

Just noticed the Keystore is in the wrong place / created wrong

See Here

Create a keystore

To create a secrets keystore, use:

metricbeat keystore create

Metricbeat creates the keystore in the directory defined by the path.data configuration setting.

Since you ran as

/usr/share/metricbeat/bin/metricbeat keystore --path.config /etc/metricbeat create

without --path.data it put it relative to the binary then the keystore is in the wrong place

Either run as shown in the docs

metricbeat keystore create

or provide the --path.data dependent on your package directory layout see here

sudo /usr/share/metricbeat/bin/metricbeat keystore create --path.config /etc/metricbeat --path.data /var/lib/metricbeat

1 Like
18

Hi @stephenb,

Thanks for quick help.

Service is running from some time now.

Created keystore as,

/usr/share/metricbeat/bin/metricbeat keystore create --path.config /etc/metricbeat --path.data /var/lib/metricbeat

It created keystore here,

-rw-------. 1 root root 190 Sep 15 06:13 /var/lib/metricbeat/metricbeat.keystore

Added key as

cat es_password | /usr/share/metricbeat/bin/metricbeat  keystore add --path.config /etc/metricbeat --path.data /var/lib/metricbeat/ ES_PWD --stdin

now service is started

systemctl status metricbeat.service 
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
   Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-09-15 06:13:40 UTC; 13min ago
     Docs: https://www.elastic.co/beats/metricbeat
 Main PID: 4536 (metricbeat)
   CGroup: /system.slice/metricbeat.service
           └─4536 /usr/share/metricbeat/bin/metricbeat --environment systemd -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /var/lib/metri

Thanks,

1 Like
19

Do you think it should be mentioned in document to use --path.config and --path.data options?

If we create keystore like below, (i.e without --path.config, --path.data options) ,
(have to be in /etc/metricbeat/ path, otherwise gives error initializing beat: error loading config file: stat metricbeat.yml: no such file or directory)

or used --path.config if not inside /etc/metricbeat (but then we might not used --path.data (as its not mentioned anywhere) and can faced this problem)

when not inside /etc/metricbeat

[root@ip-10-10-10-242 ~]#  /usr/share/metricbeat/bin/metricbeat   keystore list
error initializing beat: error loading config file: stat metricbeat.yml: no such file or directory

using --path.config option,

[root@ip-10-10-10-242 ~]#  /usr/share/metricbeat/bin/metricbeat  --path.config /etc/metricbeat  keystore list
ES_PWD
[root@ip-10-10-10-242 ~]#

Created keystore by being inside /etc/metricbeat.

[root@ip-10-10-10-242 metricbeat]# /usr/share/metricbeat/bin/metricbeat keystore create
Created metricbeat keystore
[root@ip-10-10-10-242 metricbeat]#

It creates keystores here,

[root@ip-10-10-10-242 metricbeat]# ls -l /usr/share/metricbeat/bin/data/metricbeat.keystore
-rw-------. 1 root root 130 Sep 15 06:30 /usr/share/metricbeat/bin/data/metricbeat.keystore
[root@ip-10-10-10-242 metricbeat]#

Added key

[root@ip-10-10-10-242 metricbeat]# cat /root/es_password | /usr/share/metricbeat/bin/metricbeat  keystore add  ES_PWD --stdin
Successfully updated the keystore
[root@ip-10-10-10-242 metricbeat]#

listing

[root@ip-10-10-10-242 metricbeat]#  /usr/share/metricbeat/bin/metricbeat  --path.config /etc/metricbeat keystore list
ES_PWD
[root@ip-10-10-10-242 metricbeat]#

and restarted the service but its failing to start with same previous error.

Sep 15 06:32:03 ip-10-10-10-242. metricbeat[5072]: 2021-09-15T06:32:03.241Z        ERROR        instance/beat.go:989        Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')
Sep 15 06:32:03 ip-10-10-10-242. metricbeat[5072]: Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')
Sep 15 06:32:03 ip-10-10-10-242.systemd[1]: metricbeat.service: main process exited, code=exited, status=1/FAILURE
Sep 15 06:32:03 ip-10-10-10-242. systemd[1]: Unit metricbeat.service entered failed state.
Sep 15 06:32:03 ip-10-10-10-242.systemd[1]: metricbeat.service failed.
20

@prat Glad you got it working

Actually no... the docs are correct

Did you try exactly as it is in the documentation.

When you run it as you are running it with the full path to the metricbeat binary it then is expecting all the parameters.

When you run it without the full path it is assumes the default directory layout and would have put everything in the correct place.

That's what the document shows

When I run that it creates it correctly

root@pcf-mysql-0:~$ which metricbeat
/usr/bin/metricbeat
root@pcf-mysql-0:~$ metricbeat keystore create 
Created metricbeat keystore
root@pcf-mysql-0:~$ ls -l /var/lib/metricbeat/metricbeat.keystore
-rw------- 1 root root 130 Sep 15 14:05 /var/lib/metricbeat/metricbeat.keystore
root@pcf-mysql-0:~$

:slight_smile: