# Metricbeat service is not starting after adding metricbeat keystore

**URL:** https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223
**Category:** Beats
**Tags:** metricbeat
**Created:** [September 14, 2021, 9:22pm UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223 "2021-09-14T21:22:16Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)
#### Post date: [September 14, 2021, 9:22pm UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/1 "2021-09-14T21:22:16Z")

</div>

Hi Team,

I am following below link for metricbeat keystore setup on `v7.14.0`.

> **[Metricbeat command reference | Metricbeat Reference \[master\] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/master/command-line-options.html#global-flags)**

I created the `keystore` as,

```auto
 /usr/share/metricbeat/bin/metricbeat keystore --path.config /etc/metricbeat create

```

Created the key

```auto
cat es_password | /usr/share/metricbeat/bin/metricbeat keystore --path.config /etc/metricbeat add ES_PWD --stdin

```

Able to list the key

```auto
[root@ip-10-10-10-242 metricbeat]# /usr/share/metricbeat/bin/metricbeat keystore list
ES_PWD
[root@ip-10-10-10-242 metricbeat]# 

```

Mentioned in the `metricbeat.yml` as,

```auto
setup.dashboards.enabled: true
setup.kibana:
  host: "10.10.10.242"
  username: elastic
  password: "${ES_PWD}"

```

Able to see `keystore` file.

```auto
[root@ip-10-10-10-242 metricbeat]# ls -l /usr/share/metricbeat/bin/data/metricbeat.keystore

-rw-------. 1 root root 190 Sep 14 20:54 /usr/share/metricbeat/bin/data/metricbeat.keystore

[root@ip-10-10-10-242 metricbeat]#

```

After doing above `metricbeat` service is not getting started.

```auto
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
   Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2021-09-14 21:01:52 UTC; 9min ago
     Docs: https://www.elastic.co/beats/metricbeat
  Process: 27738 ExecStart=/usr/share/metricbeat/bin/metricbeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
 Main PID: 27738 (code=exited, status=1/FAILURE)

Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: metricbeat.service: main process exited, code=exited, status=1/FAILURE
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: Unit metricbeat.service entered failed state.
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: metricbeat.service failed.
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: metricbeat.service holdoff time over, scheduling restart.
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: start request repeated too quickly for metricbeat.service
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: Failed to start Metricbeat is a lightweight shipper for metrics..
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: Unit metricbeat.service entered failed state.
Sep 14 21:01:52 ip-10-10-10-242 systemd[1]: metricbeat.service failed.

```

Can someone point out why this is happening?

Same problem is with `filebeat` also. However `heartbeat` is running fine after above configuration and heartbeat service restart.

Thanks,

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 14, 2021, 10:41pm UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/2 "2021-09-14T22:41:06Z")

</div>

Can you please check the actual Metricbeat logs, it should give you more of an indicator as to what's happening 🙂

---

<div class="post-metadata">

### Author: ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)
#### Post date: [September 15, 2021, 5:26am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/3 "2021-09-15T05:26:06Z")

</div>

all available metricbeat logs files (`/var/log/metricbeat/metricbeat`, `metricbeat.1`, ....... `metricbeat.4`) are not getting updated even after metricbeat service restart. Last log is of `2021-08-24`.

in `journalctl -fu metricbeat` can see this,

```auto
ERROR instance/beat.go:989 Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')

Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')

```

We have defined password under `kibana.yml` (output given above)

complete logs -

```auto
Sep 15 05:20:44 ip-10-10-10-242 metricbeat[3279]: 2021-09-15T05:20:44.724Z ERROR instance/beat.go:989 Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')
Sep 15 05:20:44 ip-10-10-10-242 metricbeat[3279]: Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')
Sep 15 05:20:44 ip-10-10-10-242 systemd[1]: metricbeat.service: main process exited, code=exited, status=1/FAILURE
Sep 15 05:20:44 ip-10-10-10-242 systemd[1]: Unit metricbeat.service entered failed state.
Sep 15 05:20:44 ip-10-10-10-242[1]: metricbeat.service failed.
Sep 15 05:20:45 ip-10-10-10-242[1]: metricbeat.service holdoff time over, scheduling restart.
Sep 15 05:20:45 ip-10-10-10-242[1]: start request repeated too quickly for metricbeat.service
Sep 15 05:20:45 ip-10-10-10-242[1]: Failed to start Metricbeat is a lightweight shipper for metrics..
Sep 15 05:20:45 ip-10-10-10-242[1]: Unit metricbeat.service entered failed state.
Sep 15 05:20:45 ip-10-10-10-242[1]: metricbeat.service failed.

```

Thanks,

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 15, 2021, 5:28am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/4 "2021-09-15T05:28:30Z")

</div>

Ok no worries. Can you try starting Metricbeat directly?

---

<div class="post-metadata">

### Author: ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)
#### Post date: [September 15, 2021, 5:30am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/5 "2021-09-15T05:30:42Z")

</div>

directly? Sorry didnt get you.

I am starting it with systemctl command under linux. (`systemctl restart metricbeat.service`) and it is failing to start it.

Thanks,

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 15, 2021, 5:31am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/6 "2021-09-15T05:31:47Z")

</div>

Something like `metricbeat -e` I mean, calling the binary directly.

---

<div class="post-metadata">

### Author: ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)
#### Post date: [September 15, 2021, 5:37am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/7 "2021-09-15T05:37:12Z")

</div>

ok. Thank you.

Error -

```auto
2021-09-15T05:33:43.962Z	ERROR	instance/beat.go:989	Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/bin/kibana: No directory /usr/share/metricbeat/bin/kibana/7
Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/bin/kibana: No directory /usr/share/metricbeat/bin/kibana/7

```

complete logs -

```auto
[root@ip-10-10-10-242 metricbeat]# /usr/share/metricbeat/bin/metricbeat -e
2021-09-15T05:33:42.797Z	INFO	instance/beat.go:665	Home path: [/usr/share/metricbeat/bin] Config path: [/usr/share/metricbeat/bin] Data path: [/usr/share/metricbeat/bin/data] Logs path: [/usr/share/metricbeat/bin/logs]
2021-09-15T05:33:42.798Z	INFO	instance/beat.go:673	Beat ID: 5be2653a-2dd4-4663-b475-8ce47cd2b258
2021-09-15T05:33:42.799Z	INFO	[seccomp]	seccomp/seccomp.go:101	Syscall filter could not be installed because the kernel does not support seccomp
2021-09-15T05:33:42.799Z	INFO	[beat]	instance/beat.go:1014	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat/bin", "data": "/usr/share/metricbeat/bin/data", "home": "/usr/share/metricbeat/bin", "logs": "/usr/share/metricbeat/bin/logs"}, "type": "metricbeat", "uuid": "5be2653a-2dd4-4663-b475-8ce47cd2b258"}}}
2021-09-15T05:33:42.799Z	INFO	[beat]	instance/beat.go:1023	Build info	{"system_info": {"build": {"commit": "e127fc31fc6c00fdf8649808f9421d8f8c28b5db", "libbeat": "7.14.0", "time": "2021-07-29T21:09:05.000Z", "version": "7.14.0"}}}
2021-09-15T05:33:42.799Z	INFO	[beat]	instance/beat.go:1026	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.16.6"}}}
2021-09-15T05:33:42.800Z	INFO	[beat]	instance/beat.go:1030	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-02-15T11:12:32Z","containerized":false,"name":"ip-10-0-1-142.eu-west-3.compute.internal","ip":["127.0.0.1/8","::1/128","10.10.10.242/24","fe80::4b0:5ff:fedd:ac14/64"],"kernel_version":"3.10.0-693.5.2.el7.x86_64","mac":["06:b0:05:dd:ac:14"],"os":{"type":"linux","family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":4,"patch":1708,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0,"id":"f073c429a7456b53ec3e2c53460c5c8f"}}}
2021-09-15T05:33:42.801Z	INFO	[beat]	instance/beat.go:1059	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend"],"ambient":null}, "cwd": "/etc/metricbeat", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 3574, "ppid": 3218, "seccomp": {"mode":"disabled"}, "start_time": "2021-09-15T05:33:41.650Z"}}}
2021-09-15T05:33:42.801Z	INFO	instance/beat.go:309	Setup Beat: metricbeat; Version: 7.14.0
2021-09-15T05:33:42.801Z	INFO	[publisher]	pipeline/module.go:113	Beat name: app_server
2021-09-15T05:33:42.802Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2021-09-15T05:33:42.802Z	INFO	kibana/client.go:122	Kibana url: http://10.10.10.242:5601
2021-09-15T05:33:42.804Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:105	add_cloud_metadata: hosting provider type detected as aws, metadata={"cloud":{"account":{"id":"641850179141"},"availability_zone":"eu-west-3a","image":{"id":"ami-0c60d771"},"instance":{"id":"i-03bbbf23fafc5bef9"},"machine":{"type":"t2.large"},"provider":"aws","region":"eu-west-3","service":{"name":"EC2"}}}
2021-09-15T05:33:43.337Z	INFO	kibana/client.go:122	Kibana url: http://10.10.10.242:5601
2021-09-15T05:33:43.962Z	INFO	[monitoring]	log/log.go:153	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":90,"time":{"ms":93}},"total":{"ticks":800,"time":{"ms":804},"value":800},"user":{"ticks":710,"time":{"ms":711}}},"handles":{"limit":{"hard":65535,"soft":65535},"open":10},"info":{"ephemeral_id":"cfe077f5-3ddf-42b8-9c35-d8c0baf6d7fb","uptime":{"ms":1244},"version":"7.14.0"},"memstats":{"gc_next":26976000,"memory_alloc":19665296,"memory_sys":75777032,"memory_total":118386160,"rss":137564160},"runtime":{"goroutines":20}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"system":{"cpu":{"cores":2},"load":{"1":0.09,"15":0.32,"5":0.23,"norm":{"1":0.045,"15":0.16,"5":0.115}}}}}}
2021-09-15T05:33:43.962Z	INFO	[monitoring]	log/log.go:154	Uptime: 1.246523834s
2021-09-15T05:33:43.962Z	INFO	[monitoring]	log/log.go:131	Stopping metrics logging.
2021-09-15T05:33:43.962Z	INFO	instance/beat.go:470	metricbeat stopped.
2021-09-15T05:33:43.962Z	ERROR	instance/beat.go:989	Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/bin/kibana: No directory /usr/share/metricbeat/bin/kibana/7
Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/bin/kibana: No directory /usr/share/metricbeat/bin/kibana/7
[root@ip-10-10-10-242 metricbeat]# 

```

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 15, 2021, 5:39am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/8 "2021-09-15T05:39:42Z")

</div>

Thanks! It might be useful to share your Metricbeat config file, we might be able to find why it's complaining about Kibana dashboards there.

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 15, 2021, 5:43am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/9 "2021-09-15T05:43:38Z")

</div>

I believe the issue is that the keystore is owned by root, it needs to be owned by metricbeat which is the user metricbeat is started under via systemctl

`sudo chown metricbeat:metricbeat /usr/share/metricbeat/bin/data/metricbeat.keystore`

---

<div class="post-metadata">

### Author: ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)
#### Post date: [September 15, 2021, 5:45am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/10 "2021-09-15T05:45:15Z")

</div>

If i used the password instead of key in `metricbeat.yml`, and restart, service is up and running corrently.

---

<div class="post-metadata">

### Author: ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)
#### Post date: [September 15, 2021, 5:47am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/11 "2021-09-15T05:47:26Z")

</div>

Hi @stephenb,

There is no such user `metricbeat`.

```auto
[root@ip-10-10-10-242 metricbeat]# chown metricbeat:metricbeat /usr/share/metricbeat/bin/data/metricbeat.keystore 
chown: invalid user: 'metricbeat:metricbeat'

[root@ip-10-10-10-242 metricbeat]# id metricbeat
id: metricbeat: no such user
[root@ip-10-10-10-242 metricbeat]# 

```

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 15, 2021, 5:48am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/12 "2021-09-15T05:48:13Z")

</div>

then  
`sudo chmod 644 /usr/share/metricbeat/bin/data/metricbeat.keystore`

---

<div class="post-metadata">

### Author: ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)
#### Post date: [September 15, 2021, 5:48am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/13 "2021-09-15T05:48:45Z")

</div>

Hi @warkolm,

Here is the file

```auto
[root@ip-10-10-10-242 metricbeat]# cat /etc/metricbeat/metricbeat.yml
# Ansible managed

name: app_server
fields_under_root: true
fields:
  host_id: app_server

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

setup.dashboards.enabled: true
setup.kibana:
  host: "10.10.10.242"
  username: elastic
  password: xxx
# metricbeat setup.ilm

setup.ilm.enabled: false
setup.ilm.rollover_alias: "metricbeat"
setup.ilm.pattern: "{now/d}-000001" 

output.logstash:
  hosts: ['10.10.10.242:5044']
  loadbalance: False

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
[root@ip-10-10-10-242 metricbeat]# 

```

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 15, 2021, 5:49am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/14 "2021-09-15T05:49:29Z")

</div>

Yeah sorry metricbeat install does not create a metricbeat user

If you want test run form command line the command is:

`/usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /var/lib/metricbeat --path.logs /var/log/metricbeat`

---

<div class="post-metadata">

### Author: ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)
#### Post date: [September 15, 2021, 5:55am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/15 "2021-09-15T05:55:26Z")

</div>

given 644

```auto
[root@ip-10-10-10-242 metricbeat]# ls -l /usr/share/metricbeat/bin/data/metricbeat.keystore
-rw-r--r--. 1 root root 190 Sep 14 20:54 /usr/share/metricbeat/bin/data/metricbeat.keystore

```

Giving below error while listing keystore.

```auto
[root@ip-10-10-10-242 metricbeat]# /usr/share/metricbeat/bin/metricbeat keystore list
error initializing beat: could not initialize the keystore: file ("/usr/share/metricbeat/bin/data/metricbeat.keystore") can only be writable and readable by the owner but the permissions are "-rw-r--r--" (to fix the permissions use: 'chmod go-wrx /usr/share/metricbeat/bin/data/metricbeat.keystore')
[root@ip-10-10-10-242 metricbeat]#

```

---

<div class="post-metadata">

### Author: ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)
#### Post date: [September 15, 2021, 6:01am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/16 "2021-09-15T06:01:49Z")

</div>

@stephenb,

Sorry lost your comment about directory layout. Can't see it now.

I have installed metricbeat via rpm.

```auto
[root@ip-10-10-10-242 metricbeat]# rpm -qa |grep metricbeat
metricbeat-7.14.0-1.x86_64
[root@ip-10-10-10-242 metricbeat]#

```

its saying missing field accessing `setup.kibana.password` which we have set in config file.

```auto
[root@ip-10-10-10-242 metricbeat]# /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /var/lib/metricbeat --path.logs /var/log/metricbeat
Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')
[root@ip-10-10-10-242 metricbeat]# 

```

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 15, 2021, 6:03am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/17 "2021-09-15T06:03:45Z")

</div>

Just noticed the Keystore is in the wrong place / created wrong

See [Here](https://www.elastic.co/guide/en/beats/metricbeat/current/keystore.html#creating-keystore)

### Create a keystore

To create a secrets keystore, use:

`metricbeat keystore create`

**Metricbeat creates the keystore in the directory defined by the `path.data` configuration setting.**

Since you ran as

`/usr/share/metricbeat/bin/metricbeat keystore --path.config /etc/metricbeat create`

without `--path.data` it put it relative to the binary then the keystore is in the wrong place

Either run as shown in the docs

`metricbeat keystore create`

or provide the `--path.data` dependent on your package directory layout see [here](https://www.elastic.co/guide/en/beats/metricbeat/current/directory-layout.html)

`sudo /usr/share/metricbeat/bin/metricbeat keystore create --path.config /etc/metricbeat --path.data /var/lib/metricbeat`

---

<div class="post-metadata">

### Author: ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)
#### Post date: [September 15, 2021, 6:32am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/18 "2021-09-15T06:32:41Z")

</div>

Hi @stephenb,

Thanks for quick help.

Service is running from some time now.

Created `keystore` as,

```auto
/usr/share/metricbeat/bin/metricbeat keystore create --path.config /etc/metricbeat --path.data /var/lib/metricbeat

```

It created `keystore` here,

```auto
-rw-------. 1 root root 190 Sep 15 06:13 /var/lib/metricbeat/metricbeat.keystore

```

Added key as

```auto
cat es_password | /usr/share/metricbeat/bin/metricbeat keystore add --path.config /etc/metricbeat --path.data /var/lib/metricbeat/ ES_PWD --stdin

```

now service is started

```auto
systemctl status metricbeat.service 
● metricbeat.service - Metricbeat is a lightweight shipper for metrics.
   Loaded: loaded (/usr/lib/systemd/system/metricbeat.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-09-15 06:13:40 UTC; 13min ago
     Docs: https://www.elastic.co/beats/metricbeat
 Main PID: 4536 (metricbeat)
   CGroup: /system.slice/metricbeat.service
           └─4536 /usr/share/metricbeat/bin/metricbeat --environment systemd -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /var/lib/metri

```

Thanks,

---

<div class="post-metadata">

### Author: ![prat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prat/32/79978_2.png) [@prat](https://discuss.elastic.co/u/prat)
#### Post date: [September 15, 2021, 6:41am UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/19 "2021-09-15T06:41:36Z")

</div>

Do you think it should be mentioned in document to use `--path.config` and `--path.data` options?

If we create `keystore` like below, (i.e without `--path.config`, `--path.data` options) ,  
(have to be in `/etc/metricbeat/` path, otherwise gives `error initializing beat: error loading config file: stat metricbeat.yml: no such file or directory`)

* * *

or used `--path.config` if not inside `/etc/metricbeat` (but then we might not used `--path.data` (as its not mentioned anywhere) and can faced this problem)

when not inside `/etc/metricbeat`

```auto
[root@ip-10-10-10-242 ~]# /usr/share/metricbeat/bin/metricbeat keystore list
error initializing beat: error loading config file: stat metricbeat.yml: no such file or directory

```

using `--path.config` option,

```auto
[root@ip-10-10-10-242 ~]# /usr/share/metricbeat/bin/metricbeat --path.config /etc/metricbeat keystore list
ES_PWD
[root@ip-10-10-10-242 ~]# 

```

* * *

Created `keystore` by being inside `/etc/metricbeat`.

```auto
[root@ip-10-10-10-242 metricbeat]# /usr/share/metricbeat/bin/metricbeat keystore create
Created metricbeat keystore
[root@ip-10-10-10-242 metricbeat]# 

```

It creates `keystores` here,

```auto
[root@ip-10-10-10-242 metricbeat]# ls -l /usr/share/metricbeat/bin/data/metricbeat.keystore
-rw-------. 1 root root 130 Sep 15 06:30 /usr/share/metricbeat/bin/data/metricbeat.keystore
[root@ip-10-10-10-242 metricbeat]# 

```

Added key

```auto
[root@ip-10-10-10-242 metricbeat]# cat /root/es_password | /usr/share/metricbeat/bin/metricbeat keystore add ES_PWD --stdin
Successfully updated the keystore
[root@ip-10-10-10-242 metricbeat]#

```

listing

```auto
[root@ip-10-10-10-242 metricbeat]# /usr/share/metricbeat/bin/metricbeat --path.config /etc/metricbeat keystore list
ES_PWD
[root@ip-10-10-10-242 metricbeat]# 

```

and restarted the service but its failing to start with same previous error.

```auto
Sep 15 06:32:03 ip-10-10-10-242. metricbeat[5072]: 2021-09-15T06:32:03.241Z ERROR instance/beat.go:989 Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')
Sep 15 06:32:03 ip-10-10-10-242. metricbeat[5072]: Exiting: error connecting to Kibana: missing field accessing 'setup.kibana.password' (source:'/etc/metricbeat/metricbeat.yml')
Sep 15 06:32:03 ip-10-10-10-242.systemd[1]: metricbeat.service: main process exited, code=exited, status=1/FAILURE
Sep 15 06:32:03 ip-10-10-10-242. systemd[1]: Unit metricbeat.service entered failed state.
Sep 15 06:32:03 ip-10-10-10-242.systemd[1]: metricbeat.service failed.

```

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [September 15, 2021, 2:07pm UTC](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223/20 "2021-09-15T14:07:02Z")

</div>

@prat Glad you got it working

Actually no... the docs are correct

Did you try exactly as it is in the documentation.

When you run it as you are running it with the full path to the metricbeat binary it then is expecting all the parameters.

When you run it without the full path it is assumes the default directory layout and would have put everything in the correct place.

That's what the document shows

> [@stephenb](#):
>
> Either run as shown in the docs
> 
> `metricbeat keystore create`

When I run that it creates it correctly

```auto
root@pcf-mysql-0:~$ which metricbeat
/usr/bin/metricbeat
root@pcf-mysql-0:~$ metricbeat keystore create 
Created metricbeat keystore
root@pcf-mysql-0:~$ ls -l /var/lib/metricbeat/metricbeat.keystore
-rw------- 1 root root 130 Sep 15 14:05 /var/lib/metricbeat/metricbeat.keystore
root@pcf-mysql-0:~$

```

🙂

[Next page](https://discuss.elastic.co/t/metricbeat-service-is-not-starting-after-adding-metricbeat-keystore/284223.md?page=2)
