Metricbeat unable to find keystore values with `systemctl`

[learningelastic]

I am having trouble using the /usr/share/metricbeat/bin/metricbeat keystore command to store an api key. Everytime I do a systemctl restart metricbeat.service, I get this error:

{"log.level":"error","@timestamp":"2022-11-22T23:17:39.767Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://elastic.example.com:9200)): 401 Unauthorized: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","service.name":"metricbeat","ecs.version":"1.6.0"}

This is what my /etc/metricbeat/metricbeat.yml looks like:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
path:
  data: /etc/metricbeat
  config: /etc/metricbeat
  home: /usr/share/metricbeat
  logs: /var/log/metricbeat
setup.kibana:
  host: "https://kibana.example.com:5601"
output.elasticsearch:
  hosts: ["elastic.example.com:9200"]
  protocol: "https"
  # api_key: "q9x7oYQBbqxmpKMLrnDg:ZvC-u8hFQXipvnnGox1xcw"
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

I started by running these commands:

/usr/share/metricbeat/bin/metricbeat keystore create -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /etc/metricbeat --path.logs /var/log/metricbeat


/usr/share/metricbeat/bin/metricbeat keystore add output.elasticsearch.api_key -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /etc/metricbeat --path.logs /var/log/metricbeat
# when prompted, I pasted this value in q9x7oYQBbqxmpKMLrnDg:ZvC-u8hFQXipvnnGox1xcw

This made a file /etc/metricbeat/metricbeat.keystore which has this content:

v1/0+NAiva50fTYRlZBKy76WNl/7MajMY1bGTUmMCNsJJfcoyXtSBxQ9YSVZZiyNZI3kDHaIR403AbUb8ZmNKm+VVYx3UsKCy2SUEL8RizItanPGcbIcED/tsuZyWLencyG3ccqEc8a4KRs9ujcTvcNSWJCWNrL36SFJeJbnniOi2PFHfIIPvvRh5cxIZ4FNDcWayBiCGpgBFPDlBqkt8qB2AFAZjdSq756TPMxeoMDq0/XO6P/hXS4teJSFs3iCSx8oQlZrK6

When I do systemctl restart metricbeat.service, then I get the missing authentications credentials mentioned above.

If I uncomment the line api_key: "q9x7oYQBbqxmpKMLrnDg:ZvC-u8hFQXipvnnGox1xcw" from /etc/metricbeat/metricbeat.yml, then systemctl restart metricbeat.service, then everything works fine. I see metricbeat connect to elastic and i see published data in my kibana.

How do I get metricbeat to use the keystore value for output.elasticsearch.api_key?

[stephenb (Stephen Brown)]

Check the docs here

Unfortunately, the beats keystore works a little bit different than elasticsearch you have to reference the key like an environment variable. It's not just picked up automatically by the key name like it is in elasticsearch.

output.elasticsearch.api_key: "${ES_API_KEY}"

In your case, you used the fully qualified field name. You can do that which is fine or shorten it up

[learningelastic]

Almost got it working based on your suggestion. So if I do the following:

/usr/share/metricbeat/bin/metricbeat keystore add ES_API_KEY -c /etc/metricbeat/metricbeat.yml
# then paste in q9x7oYQBbqxmpKMLrnDg:ZvC-u8hFQXipvnnGox1xcw
/usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml

THen everything works fine.

But if I do systemctl restart metricbeat.service, I get the error:

Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.api_key' (source:'/etc/metricbeat/metricbeat.yml')

I then created the file /etc/systemd/system/metricbeat.system.d/debug.conf with this content:

[Service]
Environment="BEAT_CONFIG_OPTS=-c /etc/metricbeat/metricbeat.yml"

Then I did a systemctl daemon-reload && systemctl restart metricbeat.service. But I still get the error

Exiting: error initializing publisher: missing field accessing 'output.elasticsearch.api_key' (source:'/etc/metricbeat/metricbeat.yml')

So for some reason, my systemctl can't seem to pick up the keystore variables. Any ideas?

[stephenb (Stephen Brown)]

The keystore is probably in the wrong place

Can you try without all the full paths and everything just metricbeat should be found in the path and put it in the right place..

You may need to create it as well....

metricbeat keystore create

metricbeat keystore add ES_API_KEY

[learningelastic]

I got things to work by running these two commands just now:

cp /etc/metricbeat/metricbeat.keystore /var/lib/metricbeat/
systemctl restart metricbeat.service

And this is my final /etc/metricbeat/metricbeat.yml

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.kibana:
  host: "https://kibana.example.com:5601"
output.elasticsearch:
  hosts: ["elastic.example.com:9200"]
  protocol: "https"
  api_key: "${ES_API_KEY}"
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

[stephenb (Stephen Brown)]

Yup the keystore needs to be in the data directory