I am having trouble using the /usr/share/metricbeat/bin/metricbeat keystore
command to store an api key. Everytime I do a systemctl restart metricbeat.service
, I get this error:
{"log.level":"error","@timestamp":"2022-11-22T23:17:39.767Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":150},"message":"Failed to connect to backoff(elasticsearch(https://elastic.example.com:9200)): 401 Unauthorized: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"missing authentication credentials for REST request [/]\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","service.name":"metricbeat","ecs.version":"1.6.0"}
This is what my /etc/metricbeat/metricbeat.yml
looks like:
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
path:
data: /etc/metricbeat
config: /etc/metricbeat
home: /usr/share/metricbeat
logs: /var/log/metricbeat
setup.kibana:
host: "https://kibana.example.com:5601"
output.elasticsearch:
hosts: ["elastic.example.com:9200"]
protocol: "https"
# api_key: "q9x7oYQBbqxmpKMLrnDg:ZvC-u8hFQXipvnnGox1xcw"
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
I started by running these commands:
/usr/share/metricbeat/bin/metricbeat keystore create -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /etc/metricbeat --path.logs /var/log/metricbeat
/usr/share/metricbeat/bin/metricbeat keystore add output.elasticsearch.api_key -c /etc/metricbeat/metricbeat.yml --path.home /usr/share/metricbeat --path.config /etc/metricbeat --path.data /etc/metricbeat --path.logs /var/log/metricbeat
# when prompted, I pasted this value in q9x7oYQBbqxmpKMLrnDg:ZvC-u8hFQXipvnnGox1xcw
This made a file /etc/metricbeat/metricbeat.keystore
which has this content:
v1/0+NAiva50fTYRlZBKy76WNl/7MajMY1bGTUmMCNsJJfcoyXtSBxQ9YSVZZiyNZI3kDHaIR403AbUb8ZmNKm+VVYx3UsKCy2SUEL8RizItanPGcbIcED/tsuZyWLencyG3ccqEc8a4KRs9ujcTvcNSWJCWNrL36SFJeJbnniOi2PFHfIIPvvRh5cxIZ4FNDcWayBiCGpgBFPDlBqkt8qB2AFAZjdSq756TPMxeoMDq0/XO6P/hXS4teJSFs3iCSx8oQlZrK6
When I do systemctl restart metricbeat.service
, then I get the missing authentications credentials mentioned above
.
If I uncomment the line api_key: "q9x7oYQBbqxmpKMLrnDg:ZvC-u8hFQXipvnnGox1xcw"
from /etc/metricbeat/metricbeat.yml
, then systemctl restart metricbeat.service
, then everything works fine. I see metricbeat connect to elastic and i see published data in my kibana.
How do I get metricbeat to use the keystore value for output.elasticsearch.api_key
?