AWS CloudWatch integration with Elastic using Elastic Agent

[zmoog (Maurizio Branca)]

@JypraGroup

The support staff on other case responded that API key doesn't have sufficient Priv. What will be your advice on what it should be set to?

POST /_security/api_key
{
"name": "my-api-key",
"expiration": "1d",
"role_descriptors": {
"role-a": {
"cluster":["monitor"],
"index": [
{
"names": ["logs-generic-default"],
"privileges": ["manage"]
}
]
}
}
}

Yeah, I see the log message is complaining about not having the required privileges to do the action.

The error is:

"action [indices:admin/auto_create] is unauthorized for API key id [v5R8FoIBh-MY92CQc0UC] of user [elastic/fleet-server] on indices [logs-generic-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]"

The message hints about the required privileges; TBH, this error is unexpected because the fleet server manages the agent's API keys.

[zmoog (Maurizio Branca)]

The message mentions the API key belongs to the user "elastic/fleet-server"; could you please check the privileges assigned to the service account named "elastic/fleet-server"?

You can do it by visiting Dev Tools > Console and sending this request:

GET /_security/service/elastic/fleet-server

More on this at:

Get service accounts API | Elasticsearch Guide [8.3] | Elastic

[JypraGroup (Sunil Wadhwani)]

The support has asked to change below settings on the elastic-agent-policy.yaml on the elastic agent:
From:

name: AWS_Test_Default
revision: 3
streams:

• api_sleep: 200ms
  data_stream: null
  dataset: aws.cloudwatch_logs
  ......
  To:

name: AWS_Test_Default
revision: 3
streams:

• api_sleep: 200ms
  data_stream:
  dataset: aws.cloudwatch_logs
  type: logs
  ......

What does this help with?

[JypraGroup (Sunil Wadhwani)]

Thanks for details. What should be the permission so we can make the required changes?

Thanks,

[zmoog (Maurizio Branca)]

@JypraGroup

What should be the permission so we can make the required changes?

I prefer to see current settings on your cluster before thinking about any change: can you share the output of the following request?

GET /_security/service/elastic/fleet-server

This response shouldn't contain private or sensible information, but please double-check it before sharing.

[zmoog (Maurizio Branca)]

@JypraGroup

The support has asked to change below settings on the elastic-agent-policy.yaml on the elastic agent:
From:

name: AWS_Test_Default
revision: 3
streams:

• api_sleep: 200ms
  data_stream: null
  dataset: aws.cloudwatch_logs
  ......
  To:

name: AWS_Test_Default
revision: 3
streams:

• api_sleep: 200ms
  data_stream:
  dataset: aws.cloudwatch_logs
  type: logs
  ......

What does this help with?

I'm sorry, @JypraGroup, I am unfamiliar with this setting, so the support engineer is probably in a better position to answer this question.

[JypraGroup (Sunil Wadhwani)]

Below is the response
OK (706 ms)
{
"elastic/fleet-server": {
"role_descriptor": {
"applications": [
{
"application": "kibana-",
"privileges": [
"reserved_fleet-setup"
],
"resources": [
""
]
}
],
"transient_metadata": {
"enabled": true
},
"run_as": ,
"cluster": [
"monitor",
"manage_own_api_key"
],
"indices": [
{
"privileges": [
"write",
"create_index",
"auto_configure"

[zmoog (Maurizio Branca)]

@JypraGroup

Below is the response

Hey @JypraGroup, the response looks truncated: can you double-check you copied and pasted the whole response?

As a reference, here's what I get on my test cluster:

{
  "elastic/fleet-server" : {
    "role_descriptor" : {
      "cluster" : [
        "monitor",
        "manage_own_api_key"
      ],
      "indices" : [
        {
          "names" : [
            "logs-*",
            "metrics-*",
            "traces-*",
            "synthetics-*",
            ".logs-endpoint.diagnostic.collection-*",
            ".logs-endpoint.action.responses-*"
          ],
          "privileges" : [
            "write",
            "create_index",
            "auto_configure"
          ],
          "allow_restricted_indices" : false
        },
        {
          "names" : [
            "traces-apm.sampled-*"
          ],
          "privileges" : [
            "read",
            "monitor",
            "maintenance"
          ],
          "allow_restricted_indices" : false
        },
        {
          "names" : [
            ".fleet-*"
          ],
          "privileges" : [
            "read",
            "write",
            "monitor",
            "create_index",
            "auto_configure",
            "maintenance"
          ],
          "allow_restricted_indices" : true
        }
      ],
      "applications" : [
        {
          "application" : "kibana-*",
          "privileges" : [
            "reserved_fleet-setup"
          ],
          "resources" : [
            "*"
          ]
        }
      ],
      "run_as" : [ ],
      "metadata" : { },
      "transient_metadata" : {
        "enabled" : true
      }
    }
  }
}

The most interesting section is the elastic/fleet-server.role_descriptor.indices list:

{
    "names": [
        "logs-*",
        "metrics-*",
        "traces-*",
        "synthetics-*",
        ".logs-endpoint.diagnostic.collection-*",
        ".logs-endpoint.action.responses-*"
    ],
    "privileges": [
        "write",
        "create_index",
        "auto_configure"
    ],
    "allow_restricted_indices": false
}