AWS CloudWatch integration with Elastic using Elastic Agent

Elastic Stack Beats
,
1

Hi all,
I have Elastic agent installed on the endpoint and I can see the logs coming in. The policy has AWS CloudWatch integration however I am not sure what else is required to get the logs and metrics flowing from the AWS Cloudwatch into Elastic. It seems like there are few options (access keys ans IAM role) however, the permissions required is not clear.

2

Required permissions are described in the integration documentation -

#### AWS Permissions

Specific AWS permissions are required for the IAM user to make specific AWS API calls. In order to enable AWS integration to collect metrics and logs from all supported service, please make sure these permissions are given:

* ec2:DescribeInstances
* ec2:DescribeRegions
* cloudwatch:GetMetricData
* cloudwatch:ListMetrics
* iam:ListAccountAliases
* rds:DescribeDBInstances
* rds:ListTagsForResource
* s3:GetObject
* sns:ListTopics
* sqs:ChangeMessageVisibility
* sqs:DeleteMessage
* sqs:ListQueues
* sqs:ReceiveMessage
* sts:AssumeRole
* sts:GetCallerIdentity
* tag:GetResources
3

Hey @JypraGroup, welcome to the Elastic community!

As @Guncixx already suggested, the best place to start is probably the integrations docs:

The docs team recently updated and improved this page, so if you have visited it in the past, this is a perfect moment to reread it.

The doc describes all your authentication options, like using the access key directly, IAM roles, and others. To learn more, check out the AWS Credentials section.

The most up-to-date list of permissions required is available in the AWS Permissions section.

4

Thanks for the warm welcome!

With the access key option, it’s unclear what permission needs to be assigned to the user for the integration to work.

The arn role option has the sts:assume role assigned, AWS recommends to assign minimum permissions however, the minimum required permissions are not listed in the doc.

Thanks,

5

IMO we have two problems to solve here:

  1. Which authentication method to use;
  2. What is the minimum set of permission required for the services we want to use.

As a general rule, I suggest starting small with something simple to make it work and then iterating until we reach an optimal solution.

For problem 1: the most straightforward option is to use the access and secret keys.

If you already have an IAM user, create a new IAM policy with all the permissions listed in the docs and attach the policy to the IAM user.

Now, by using the access key and secret key for the IAM user in the integration settings, you will be able to use all supported services.

For problem 2: unfortunately, the docs do not list the required permissions for each service.

Let me know which integration/service you want to use, and I'll help you find the minimal permission list for your use case.

We can use this work to update the docs for our future selves and other users.

6

We will use the access key method. We would need minimum permissions required for the integration to work which we have assign during user creation.

permissions
permissions1341×837 44.2 KB

7

Great! Which integration(s) are you planning to use?

Screenshots from the integration settings are okay, so I know which one you plan to use, and I can build a tailored permission list.

8

Hi,

I am planning to integrate with AWS CloudWatch.

Regards,

9

Hey @JypraGroup, if you're going to use CloudWatch metrics and logs. here's the IAM policy tailored to support both CloudWatch Metrics and Logs:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricData",
                "cloudwatch:ListMetrics",
                "iam:ListAccountAliases",
                "ec2:DescribeRegions",
                "tag:GetResources",
                "logs:FilterLogEvents",
                "logs:DescribeLogGroups",
                "sts:GetCallerIdentity"
            ],
            "Resource": "*"
        }
    ]
}

I forgot to mention that Filebeat and Metricbeat [1] documentation lists the permissions required for each module (and then integration):

  1. The current version of the Elastic Agent orchestrates both Metricbeat and Filebeat behind the scenes to get its job done. This is going to change in future releases. ↩︎

1 Like
10

Hi,

Thanks for the details.

What permissions needs to be assigned to the user who’s access and secret keys will be used within the configuration?

11

Hi Zmoog, If we have to use the access key and ARN role method, we need to associate permissions with the user when generating the keys. What those permissions will be? The ARN role that we will be using, I believe the above permissions needs to be associated with that role, is that correct?

12

Hi Zmoog, the permission set are different in different documents AWS | Elastic Documentation and AWS cloudwatch metricset | Metricbeat Reference [8.3] | Elastic and we have tried both and combined and still we can't see CloudWatch logs in Elastic.

13

There is known bug related to CloudWatch integration [AWS] CloudWatch logs integration fails with custom namespace and dataset · Issue #3112 · elastic/integrations · GitHub

14

@JypraGroup are you using a custom namespace and dataset, as described in the issue?

15

Okay, let's see if we can understand what's going on with your setup.

Can you share the logs from the Agent and Filebeat running behind the scenes?

16

If you want to use an IAM role with the actual permissions, the IAM user with the access/secret key only needs the permissions to assume the role.

Let me know if you are familiar with this process, otherwise, I can find an example to use as a starting point for your configuration.

17

We were using Custom namespace but switched to default as per the advice.

18

Example will be good, thanks.

19

We are getting the below error after using the default namespace.

HI

i am still getting the error could you please check i have created the integration in default space and configured but i still see the same error

18:19:24.155
elastic_agent.filebeat
[elastic_agent.filebeat][warn] Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.July, 13, 19, 26, 33, 0, time.UTC), Meta:{"_id":"36968846118257707368218283182713149301655432061887250432","raw_index":"logs-generic-default"}, Fields:{"agent":{"ephemeral_id":"f623e4f9-2bec-4b49-b117-2e59c66f399f","id":"423d1d45-ca8b-422a-81b5-ed12efccc3ba","name":"ip-172-31-18-17.ec2.internal","type":"filebeat","version":"8.1.2"},"awscloudwatch":{"ingestion_time":"2022-07-13T19:26:39.000Z","log_group":"/var/log/httpd/access_log","log_stream":"i-01d3e0069c2f827e9"},"cloud":{"provider":"aws","region":"us-east-1"},"data_stream":{"dataset":"generic","namespace":"default","type":"logs"},"ecs":{"version":"8.0.0"},"elastic_agent":{"id":"423d1d45-ca8b-422a-81b5-ed12efccc3ba","snapshot":false,"version":"8.1.2"},"event":{"dataset":"generic","id":"36968846118257707368218283182713149301655432061887250432","ingested":"2022-07-19T12:49:22.875Z"},"input":{"type":"aws-cloudwatch"},"log.file.path":"/var/log/httpd/access_log/i-01d3e0069c2f827e9","message":"127.0.0.1 - - [13/Jul/2022:19:26:33 +0000] "GET /phpinfo.php HTTP/1.1" 200 95009 "-" "curl/7.79.1"","tags":["forwarded","aws-cloudwatch-logs"]}, Private:(*aws.EventACKTracker)(0xc000b614d0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [v5R8FoIBh-MY92CQc0UC] of user [elastic/fleet-server] on indices [logs-generic-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]"}, dropping event! 18:19:24.155 elastic_agent.filebeat [elastic_agent.filebeat][warn] Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.July, 13, 19, 26, 48, 0, time.UTC), Meta:{"_id":"36968846452858088326971752816521732839679088614004883456","raw_index":"logs-generic-default"}, Fields:{"agent":{"ephemeral_id":"f623e4f9-2bec-4b49-b117-2e59c66f399f","id":"423d1d45-ca8b-422a-81b5-ed12efccc3ba","name":"ip-172-31-18-17.ec2.internal","type":"filebeat","version":"8.1.2"},"awscloudwatch":{"ingestion_time":"2022-07-13T19:26:54.000Z","log_group":"/var/log/httpd/access_log","log_stream":"i-01d3e0069c2f827e9"},"cloud":{"provider":"aws","region":"us-east-1"},"data_stream":{"dataset":"generic","namespace":"default","type":"logs"},"ecs":{"version":"8.0.0"},"elastic_agent":{"id":"423d1d45-ca8b-422a-81b5-ed12efccc3ba","snapshot":false,"version":"8.1.2"},"event":{"dataset":"generic","id":"36968846452858088326971752816521732839679088614004883456","ingested":"2022-07-19T12:49:22.875Z"},"input":{"type":"aws-cloudwatch"},"log.file.path":"/var/log/httpd/access_log/i-01d3e0069c2f827e9","message":"127.0.0.1 - - [13/Jul/2022:19:26:48 +0000] "GET /phpinfo.php HTTP/1.1" 200 95009 "-" "curl/7.79.1"","tags":["forwarded","aws-cloudwatch-logs"]}, Private:(*aws.EventACKTracker)(0xc000b614d0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id [v5R8FoIBh-MY92CQc0UC] of user [elastic/fleet-server] on indices [logs-generic-default], this action is granted by the index privileges [auto_configure,create_index,mana

20

The support staff on other case responded that API key doesn't have sufficient Priv. What will be your advice on what it should be set to?

POST /_security/api_key
{
"name": "my-api-key",
"expiration": "1d",
"role_descriptors": {
"role-a": {
"cluster":["monitor"],
"index": [
{
"names": ["logs-generic-default"],
"privileges": ["manage"]
}
]
}
}
}