Thanks for reaching out - I think I am making progress. I am running 8.8.1 elasticsearch and beats agents. Installed using APT. sysctrl starts metricbeat service on a cluster node.
I am starting metricbeat using api value from key store.
Mostly trying to learn how to fish - Does the /modules.d/elasticsearch-xpack.yml override the logon credentials stored in the metricbeat.yml?
Is the module loaded into a separate running service? If not, I was thinking about combining to a single api key and use the same credential in the api_key for both yaml files.
Using the template from the elastic doc, built the following:
POST /_security/api_key
{
"name": "Cluster_Node_Beat_Agents",
"expiration": "3600d",
"role_descriptors": {
"Metricbeat_Publish": {
"cluster": [
"monitor",
"read_ilm",
"read_pipeline"
],
"index": [
{
"names": [
"metricbeat-*"
],
"privileges": [
"view_index_metadata",
"create_doc"
]
}
]
},
"Node_Based_Cluster_Monitoring": {
"cluster": [
"manage_index_templates",
"manage_ingest_pipelines",
"monitor",
"cluster:admin/ilm/get",
"cluster:admin/ilm/put",
"cluster:monitor/xpack/watcher/watch/get",
"cluster:admin/xpack/watcher/watch/put",
"cluster:admin/xpack/watcher/watch/delete"
],
"indices": [
{
"names": [
".monitoring-*"
],
"privileges": [
"all"
],
"allow_restricted_indices": false
},
{
"names": [
"metricbeat-*"
],
"privileges": [
"index",
"create_index",
"view_index_metadata",
"indices:admin/aliases",
"indices:admin/rollover"
],
"allow_restricted_indices": false
}
]
}
}
}
However, I then get the error:
Jul 17 10:35:00 dtcepvp-elkna metricbeat[25159]: {"log.level":"error","@timestamp":"2023-07-17T10:35:00.211-0700","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.index: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}
So, it seems the remote_monitoring_agent role has something more that I cannot add to my api_key to get it to work.
I think I am just being stubborn, I would prefer to use an api key, but maybe should just use username and password for agent logons...