General Log Warning Question

[Ray3]

I deployed metricbeat to a node. Unless I use superuser role, I will get warnings in the log, that it cannot take certain actions as the api key used is unauthorized. I do see data reported in kibana, it appears metrics are getting logged.

The question is, are some warning normal? Or should I keep adding privilages until all I have are info messages?

Thanks

[stephenb (Stephen Brown)]

Hi @Ray3 welcome to the community!

We need a bot more detail to help.

What Version?

How did you install?

How are you starting metricbeat?

Could you share your metricbeat.yml and any other modules you have enabled?

And we need to see the actual log message

[Ray3]

Thanks for reaching out - I think I am making progress. I am running 8.8.1 elasticsearch and beats agents. Installed using APT. sysctrl starts metricbeat service on a cluster node.

I am starting metricbeat using api value from key store.

Mostly trying to learn how to fish - Does the /modules.d/elasticsearch-xpack.yml override the logon credentials stored in the metricbeat.yml?

Is the module loaded into a separate running service? If not, I was thinking about combining to a single api key and use the same credential in the api_key for both yaml files.

Using the template from the elastic doc, built the following:

POST /_security/api_key
{
  "name": "Cluster_Node_Beat_Agents",
  "expiration": "3600d",   
  "role_descriptors": { 
    "Metricbeat_Publish": {
      "cluster": [
        "monitor",
        "read_ilm", 
        "read_pipeline"
        ],
      "index": [
        {
          "names": [
            "metricbeat-*"
            ],
          "privileges": [
            "view_index_metadata", 
            "create_doc"
            ]
        }
      ]
    },
    "Node_Based_Cluster_Monitoring": {
      "cluster": [
         "manage_index_templates",
         "manage_ingest_pipelines",
         "monitor",
         "cluster:admin/ilm/get",
         "cluster:admin/ilm/put",
         "cluster:monitor/xpack/watcher/watch/get",
         "cluster:admin/xpack/watcher/watch/put",
         "cluster:admin/xpack/watcher/watch/delete"
        ],
      "indices": [
        {
          "names": [
            ".monitoring-*"
            ],
          "privileges": [
            "all"
            ],
            "allow_restricted_indices": false
        },
        {
          "names": [
            "metricbeat-*"
            ],
          "privileges": [
            "index",
            "create_index",
            "view_index_metadata",
            "indices:admin/aliases",
            "indices:admin/rollover"
          ],
          "allow_restricted_indices": false
        }
      ]
    }
  }
}

However, I then get the error:
Jul 17 10:35:00 dtcepvp-elkna metricbeat[25159]: {"log.level":"error","@timestamp":"2023-07-17T10:35:00.211-0700","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.index: error determining if connected Elasticsearch node is master: HTTP error 401 in : 401 Unauthorized","service.name":"metricbeat","ecs.version":"1.6.0"}

So, it seems the remote_monitoring_agent role has something more that I cannot add to my api_key to get it to work.

I think I am just being stubborn, I would prefer to use an api key, but maybe should just use username and password for agent logons...

[stephenb (Stephen Brown)]

@Ray3

Mostly trying to learn how to fish - Does the /modules.d/elasticsearch-xpack.yml override the logon credentials stored in the metricbeat.yml?

Is the module loaded into a separate running service? If not, I was thinking about combining to a single api key and use the same credential in the api_key for both yaml files.

Hi @Ray3 Think perhaps there is some common confusion:

1. metricbeat needs and elasticsearch cluster to talk to, that is what is in the output section of the metrcbeat.yml

2. If you are monitoring and elasticsearch cluster which may or MAY NOT be (which is quite often the case) the same cluster you are sending the metrics to you need credentials / API key for that.

3. Yes you can put the module directly into the metricbeat.yml if you please. No modules are not a separate process space. (could be multiple threads though)

Hope this helps

[Ray3]

Thanks again Stephen,

I have tested creating a user account with proper roles - and all is well if I configure both yml files to logon by username and password. however, as soon as I create an API key, cluster monitoring begins to show errors.

So taking a working username and password, tried to create an api key for logon for those credentials such as in the example:

POST /_security/api_key/grant
{
  "grant_type": "password",
  "username": "Cluster_Node_Beats_Agents",
  "password": "testtest",
  "api_key" : {
    "name": "Cluster_Node_Beats_Agents"
  }
}

But then the errors return. It looks like for cluster monitoring, (remote_monitoring_agent role) the modules yml file, is not happy with an api_key, no matter how I go about it.

I get the sense most are just using username and password and then protecting those values in the key store when monitoring a cluster with a beats agent?

[stephenb (Stephen Brown)]

I think it is a limitation...

I tried with a super user api_key and could not get it to work.... asking internally... agree seems odd.

EDIT Yup limitation that looks like it is going to be addressed in an upcoming release you can track it here

[Stack Monitoring] Add missing metricbeat api_key configuration to elasticsearch integration module

[Ray3]

Ok thanks Stephen! Very helpful!