# Metricbeat unable to find keystore values with \`systemctl\`
**URL:** https://discuss.elastic.co/t/metricbeat-unable-to-find-keystore-values-with-systemctl/319595
**Category:** Beats
**Tags:** metricbeat
**Created:** [November 22, 2022, 11:28pm UTC](https://discuss.elastic.co/t/metricbeat-unable-to-find-keystore-values-with-systemctl/319595 "2022-11-22T23:28:23Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@learningelastic](https://discuss.elastic.co/u/learningelastic)
#### Post date: [November 22, 2022, 11:28pm UTC](https://discuss.elastic.co/t/metricbeat-unable-to-find-keystore-values-with-systemctl/319595/1 "2022-11-22T23:28:23Z")
I am having trouble using the `/usr/share/metricbeat/bin/metricbeat keystore` command to store an api key. Everytime I do a `systemctl restart metricbeat.service`, I get this error:
```auto
{"log.level":"error","@timestamp":"2022-11-22T23:17:39.767Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://elastic.example.com:9200)): 401 Unauthorized: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","service.name":"metricbeat","ecs.version":"1.6.0"}
```
This is what my `/etc/metricbeat/metricbeat.yml` looks like:
```auto
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
path:
data: /etc/metricbeat
config: /etc/metricbeat
home: /usr/share/metricbeat
logs: /var/log/metricbeat
setup.kibana:
host: "https://kibana.example.com:5601"
output.elasticsearch:
hosts: ["elastic.example.com:9200"]
protocol: "https"
# api_key: "q9x7oYQBbqxmpKMLrnDg:ZvC-u8hFQXipvnnGox1xcw"
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
```
I started by running these commands:
```auto
/usr/share/metricbeat/bin/metricbeat keystore create -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /etc/metricbeat --path.logs /var/log/metricbeat
/usr/share/metricbeat/bin/metricbeat keystore add output.elasticsearch.api_key -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /etc/metricbeat --path.logs /var/log/metricbeat
# when prompted, I pasted this value in q9x7oYQBbqxmpKMLrnDg:ZvC-u8hFQXipvnnGox1xcw
```
This made a file `/etc/metricbeat/metricbeat.keystore` which has this content:
```auto
v1/0+NAiva50fTYRlZBKy76WNl/7MajMY1bGTUmMCNsJJfcoyXtSBxQ9YSVZZiyNZI3kDHaIR403AbUb8ZmNKm+VVYx3UsKCy2SUEL8RizItanPGcbIcED/tsuZyWLencyG3ccqEc8a4KRs9ujcTvcNSWJCWNrL36SFJeJbnniOi2PFHfIIPvvRh5cxIZ4FNDcWayBiCGpgBFPDlBqkt8qB2AFAZjdSq756TPMxeoMDq0/XO6P/hXS4teJSFs3iCSx8oQlZrK6
```
When I do `systemctl restart metricbeat.service`, then I get the `missing authentications credentials mentioned above`.
If I uncomment the line `api_key: "q9x7oYQBbqxmpKMLrnDg:ZvC-u8hFQXipvnnGox1xcw"` from `/etc/metricbeat/metricbeat.yml`, then `systemctl restart metricbeat.service`, then everything works fine. I see metricbeat connect to elastic and i see published data in my kibana.
How do I get metricbeat to use the keystore value for `output.elasticsearch.api_key`?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 22, 2022, 11:55pm UTC](https://discuss.elastic.co/t/metricbeat-unable-to-find-keystore-values-with-systemctl/319595/2 "2022-11-22T23:55:39Z")
Check the docs [here](https://www.elastic.co/guide/en/beats/metricbeat/current/keystore.html)
Unfortunately, the beats keystore works a little bit different than elasticsearch you have to reference the key like an environment variable. It's not just picked up automatically by the key name like it is in elasticsearch.
`output.elasticsearch.api_key: "${ES_API_KEY}"`
In your case, you used the fully qualified field name. You can do that which is fine or shorten it up
---
### Author:  [@learningelastic](https://discuss.elastic.co/u/learningelastic)
#### Post date: [November 23, 2022, 12:23am UTC](https://discuss.elastic.co/t/metricbeat-unable-to-find-keystore-values-with-systemctl/319595/3 "2022-11-23T00:23:39Z")
Almost got it working based on your suggestion. So if I do the following:
```auto
/usr/share/metricbeat/bin/metricbeat keystore add ES_API_KEY -c /etc/metricbeat/metricbeat.yml
# then paste in q9x7oYQBbqxmpKMLrnDg:ZvC-u8hFQXipvnnGox1xcw
/usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml
```
THen everything works fine.
But if I do `systemctl restart metricbeat.service`, I get the error:
```auto
Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.api_key' (source:'/etc/metricbeat/metricbeat.yml')
```
I then created the file `/etc/systemd/system/metricbeat.system.d/debug.conf` with this content:
```auto
[Service]
Environment="BEAT_CONFIG_OPTS=-c /etc/metricbeat/metricbeat.yml"
```
Then I did a `systemctl daemon-reload && systemctl restart metricbeat.service`. But I still get the error
```auto
Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.api_key' (source:'/etc/metricbeat/metricbeat.yml')
```
So for some reason, my systemctl can't seem to pick up the keystore variables. Any ideas?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 23, 2022, 12:58am UTC](https://discuss.elastic.co/t/metricbeat-unable-to-find-keystore-values-with-systemctl/319595/4 "2022-11-23T00:58:16Z")
The keystore is probably in the wrong place
Can you try without all the full paths and everything just metricbeat should be found in the path and put it in the right place..
You may need to create it as well....
`metricbeat keystore create`
`metricbeat keystore add ES_API_KEY`
---
### Author:  [@learningelastic](https://discuss.elastic.co/u/learningelastic)
#### Post date: [November 23, 2022, 4:08am UTC](https://discuss.elastic.co/t/metricbeat-unable-to-find-keystore-values-with-systemctl/319595/5 "2022-11-23T04:08:07Z")
I got things to work by running these two commands just now:
```auto
cp /etc/metricbeat/metricbeat.keystore /var/lib/metricbeat/
systemctl restart metricbeat.service
```
And this is my final `/etc/metricbeat/metricbeat.yml`
```auto
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
setup.kibana:
host: "https://kibana.example.com:5601"
output.elasticsearch:
hosts: ["elastic.example.com:9200"]
protocol: "https"
api_key: "${ES_API_KEY}"
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 23, 2022, 4:09am UTC](https://discuss.elastic.co/t/metricbeat-unable-to-find-keystore-values-with-systemctl/319595/6 "2022-11-23T04:09:42Z")
Yup the keystore needs to be in the data directory
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 21, 2022, 6:09am UTC](https://discuss.elastic.co/t/metricbeat-unable-to-find-keystore-values-with-systemctl/319595/7 "2022-12-21T06:09:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.