Elasticsearch missing authentication credentials for REST request and won't let me setup elasticsearch passwords

Elastic Stack Elasticsearch
1

I deployed Elasticsearch on Kubernetes and its running but I get these errors in my logs:

"message":"monitoring execution failed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-data-0][generic][T#1]","log.logger":"org.elasticsearch.xpack.monitoring.MonitoringService","elasticsearch.cluster.uuid":"ggc2JOEnQ-mJuYxcCvzNOQ","elasticsearch.node.id":"0CY571uHRiy2J9Sm3dXQzg","elasticsearch.node.name":"elasticsearch-data-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.xpack.monitoring.exporter.ExportException","error.message":"failed to flush export bulks"

"message":"unexpected error while indexing monitoring document", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-data-0][generic][T#1]","log.logger":"org.elasticsearch.xpack.monitoring.exporter.local.LocalExporter","elasticsearch.cluster.uuid":"ggc2JOEnQ-mJuYxcCvzNOQ","elasticsearch.node.id":"0CY571uHRiy2J9Sm3dXQzg","elasticsearch.node.name":"elasticsearch-data-0","elasticsearch.cluster.name":"elasticsearch","error.type":"org.elasticsearch.xpack.monitoring.exporter.ExportException","error.message":"org.elasticsearch.action.UnavailableShardsException: [.monitoring-es-7-2024.05.20][0] primary shard is not active Timeout: [1m]

And when I try to run curl request on my Elasticsearch pod I get this error:

"missing authentication credentials for REST request [/_cluster/stats?pretty]"

When I try to setup passwords it gives me this error:

Running with configuration path: /usr/share/elasticsearch/config

Testing if bootstrap password is valid for http://192.168.225.106:9200/_security/_authenticate?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "unable to authenticate user [elastic] for REST request [/_security/_authenticate?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Basic realm=\"security\" charset=\"UTF-8\"",
            "ApiKey"
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "unable to authenticate user [elastic] for REST request [/_security/_authenticate?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Basic realm=\"security\" charset=\"UTF-8\"",
        "ApiKey"
      ]
    }
  },
  "status" : 401
}

Failed to authenticate user 'elastic' against http://192.168.225.106:9200/_security/_authenticate?pretty
Possible causes include:
 * The password for the 'elastic' user has already been changed on this cluster
 * Your elasticsearch node is running against a different keystore
   This tool used the keystore at /usr/share/elasticsearch/config/elasticsearch.keystore

You can use the `elasticsearch-reset-password` CLI tool to reset the password of the 'elastic' user


ERROR: Failed to verify bootstrap password

Whan can cause this and how to resolve it?

2

Did you check those possibilities?

Did you try this?

3

Yes, and I got this Error:

WARNING: Owner of file [/usr/share/elasticsearch/config/users] used to be [root], but now is [elasticsearch]
WARNING: Owner of file [/usr/share/elasticsearch/config/users_roles] used to be [root], but now is [elasticsearch]
Failed to determine the health of the cluster. Cluster health is currently RED.
This means that some cluster data is unavailable and your cluster is not fully functional.
The cluster logs (https://www.elastic.co/guide/en/elasticsearch/reference/8.7/logging.html) might contain information/indications for the underlying cause
It is recommended that you resolve the issues with your cluster before continuing
It is very likely that the command will fail when run against an unhealthy cluster.

If you still want to attempt to execute this command against an unhealthy cluster, you can pass the `-f` parameter.

ERROR: Failed to determine the health of the cluster. Cluster health is currently RED.

And when I checked the logs in Elasticsearch I had this Error:

{"@timestamp":"2024-05-22T14:43:04.713Z", "log.level":"ERROR", "message":"security index is unavailable. short circuiting retrieval of user [autogenerated_RWA7n9Yc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-data-0][transport_worker][T#4]","log.logger":"org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore","elasticsearch.cluster.uuid":"YspupsdGSt6JN-O76fTOmw","elasticsearch.node.id":"ZXXFFxK7S6GhXACvLLg3fw","elasticsearch.node.name":"elasticsearch-data-0","elasticsearch.cluster.name":"elasticsearch"}```

I am stuck at this for a week and can't solve it in any way
4

You don't have a security problem, you have a cluster health problem.

Your cluster is missing data, you'll need to check the logs for more details.