401 after updating API Key role descriptors

Elastic Stack Elasticsearch
,
1

Hello,

I'm trying to limit default privileges of the API key when it's being created. By default it's created by terraform with superuser account which I feel has too much access.

We want to use the API_Key to connect to Elasticsearch from the .NET application. Application will run querying, indexing operations, it will create, delete indexes, should also be able to monitor cluster health connectivity.

Now by default if I just create the API key with permissions of the super user this is what we get.
With these permissions the applications works just fine

{
  "api_keys": [
    {
      "id": "",
      "name": "",
      "creation": ,
      "invalidated": ,
      "username": "",
      "realm": "found",
      "metadata": {},
      "role_descriptors": {},
      "limited_by": [
        {
          "superuser": {
            "cluster": [
              "all"
            ],
            "indices": [
              {
                "names": [
                  "*"
                ],
                "privileges": [
                  "all"
                ],
                "allow_restricted_indices": false
              },
              {
                "names": [
                  "*"
                ],
                "privileges": [
                  "monitor",
                  "read",
                  "view_index_metadata",
                  "read_cross_cluster"
                ],
                "allow_restricted_indices": true
              }
            ],
            "applications": [
              {
                "application": "*",
                "privileges": [
                  "*"
                ],
                "resources": [
                  "*"
                ]
              }
            ],
            "run_as": [
              "*"
            ],
            "metadata": {
              "_reserved": true
            },
            "transient_metadata": {
              "enabled": true
            }
          }
        }
      ]
    }
  ]
}

Now when I changed the role_descriptors to limit access of the key it looks like this

{
  "api_keys": [
    {
      "id": "",
      "name": "",
      "creation": ,
      "invalidated": ,
      "username": "",
      "realm": "found",
      "metadata": {},
      "role_descriptors": {
        "role-a": {
          "cluster": [
            "monitor"
          ],
          "indices": [
            {
              "names": [
                "*"
              ],
              "privileges": [
                "all"
              ],
              "allow_restricted_indices": false
            }
          ],
          "applications": [],
          "run_as": [],
          "metadata": {},
          "transient_metadata": {
            "enabled": true
          }
        }
      }
    }
  ]
}

But now we get 401 error

System.InvalidOperationException : Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration. Call: Status code 401

Now since I left all of the indices permissions with all privileges I'm not sure which part if the one that is causing the issues. I feel like the key doesn't need most of the Cluster privileges since it causes security vulnerability.

2

Hi @Mateusz_Migala

What version of the stack are you using?

And what call / API are you sending when you get the error, how and where are you using it?

It's always very helpful if you actually show the command you're running and then the output, otherwise we're just guessing.

When I look at the role you only included one role of monitoring which is very limited. So what are you trying to accomplish??

3

We are using 8.6.1 version of elastic.

So I kind of went over the use case I thought. It seems all operations started failing. Search, Indexing, index creation. They are run from .NET app using API key to connect to the cluster.

but one example would be index creation.

        public async Task<CreateIndexResponse> IndexCreateAsync(IndexName index, Func<CreateIndexDescriptor, ICreateIndexRequest> selector = null, CancellationToken ct = default)
        {
            return await _elasticClient.Indices.CreateAsync(index, selector, ct);
        }

and this is how the connection part looks like

        private static ElasticClient CreateElasticClient(IConfiguration configuration)
        {
            string apiKey = configuration[ConfigurationConstants.ElasticApiKey];
            string elasticPrivateLinkUrl = configuration[ConfigurationConstants.ElasticPrivateLinkUrl];

            ConnectionSettings settings =
                new ConnectionSettings(new Uri(elasticPrivateLinkUrl))
                    .ApiKeyAuthentication(new ApiKeyAuthenticationCredentials(apiKey))
                    .DefaultIndex(IndexingConstants.DefaultMdcId)
                    .EnableDebugMode()
                    .MaximumRetries(ElasticsearchIndexSettings.MaxRetries)
                    .MaxRetryTimeout(ElasticsearchIndexSettings.MaxRetryTimeout)
                    .EnableApiVersioningHeader();

            return new ElasticClient(settings);

So I'm not sure which privilege I removed is causing the issue.

Yes I included only Monitoring. What would be the cluster role that is needed to do the operations I mentioned ?

4

So I would test using CURL or postman first to eliminate the variables.

Are you using SHEILD plugin or something I do not recognize that error... Shield is ancient... maybe that is an old error message... but I searched the code and I do not see that error message.

Are you only using core Elastic Components?

5

Well we are using PrivateLink for connections. I don't know anything about Shield . I can try some testing using Postman. Trying to get to the bottom of it. I thought someone might know something I'm missing right of the bat.

6

The weird part is you should get a very descriptive error saying that whatever you're trying to do requires role A, B and C.

That's the normal error message when you try to do something you don't have authorization for.

I haven't seen that error before.

I would definitely try to curl or postman furst to see what's going on.. just POST a simple document to s new index.

7

Maybe the error is specific to NEST ? it's the .NET elastic client we are using.

Also I just did few quick tests and if I manually create the API key with the permissions I mentioned everything works fine.

POST /_security/api_key
{
  "name": "test",
  "role_descriptors": { 
    "role-a": {
      "cluster": ["monitor"],
      "index": [
        {
          "names": ["*"],
          "privileges": ["all"]
        }
      ]
    }
  }
}

The issue started occurring when I tried to set the same set of permissions using terraform.

# Create Elastic API KEY
resource "elasticstack_elasticsearch_security_api_key" "api_key" {
  name = "elastic_api_key"

    role_descriptors = jsonencode({
    role-a = {
      cluster = ["monitor"],
      indices = [
        {
          names      = ["*"],
          privileges = ["all"]
        }
      ]
    }
  })

  elasticsearch_connection {
    endpoints = [coalesce(var.ELASTICPRIVATELINKURL, "${ec_deployment.elasticsearch.elasticsearch[0].https_endpoint}")]
    username  = ec_deployment.elasticsearch.elasticsearch_username
    password  = ec_deployment.elasticsearch.elasticsearch_password
  }
  
    lifecycle {
      ignore_changes = [
        elasticsearch_connection[0].endpoints[0]
      ]
  }  
}

but I just verified using dev console and if I run

GET /_security/api_key?id=nameOfKey

both keys are identical. The only difference being username and realm since I created the key using my account and not elastic superuser account. Also one key was just created using Post and the second one was Post and then Put (but i dont see how that makes any difference)

8

Ok I did one more quick check

This error is always shown if the API key is incorrect

        private static ElasticClient CreateElasticClient(IConfiguration configuration)
        {
            string apiKey = configuration[ConfigurationConstants.ElasticApiKey];
            string elasticPrivateLinkUrl = configuration[ConfigurationConstants.ElasticPrivateLinkUrl];

            ConnectionSettings settings =
                new ConnectionSettings(new Uri(elasticPrivateLinkUrl))
                    .ApiKeyAuthentication(new ApiKeyAuthenticationCredentials("KeyThatDoesntExist"))
                    .DefaultIndex(IndexingConstants.DefaultMdcId)
                    .EnableDebugMode()
                    .MaximumRetries(ElasticsearchIndexSettings.MaxRetries)
                    .MaxRetryTimeout(ElasticsearchIndexSettings.MaxRetryTimeout)
                    .EnableApiVersioningHeader();

            return new ElasticClient(settings);
        }

will cause

System.InvalidOperationException : Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration. Call: Status code 401 from: POST /temp-index-for-integration-tests-add3cafd-fb59-4de2-93ad-e716b24749ce/_search?pretty=true&error_trace=true&typed_keys=true. ServerError: Type: security_exception Reason: "unable to authenticate with provided credentials and anonymous access is not allowed for this request"

Still not sure though why I started to see this error for the key with updated roles. Maybe I will try to rerun the terraform script again since the roles seem to be ok.

9

Ok I found what the issue was...

We keep Elastic API key reference in Azure Key vault and Azure App config.

Azure app config wasnt correctly updated. Will need to look into the reason why.

Anyway the error I mentioned is basically saying API key doesnt exist.

I've seen 403s when there was something wrong with actual permissions

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.