# AWS CloudWatch integration with Elastic using Elastic Agent **URL:** \ **Category:** Beats\ **Tags:** elastic-agent, integrations\ **Created:** [June 28, 2022, 5:42am UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318 "2022-06-28T05:42:12Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![JypraGroup](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JypraGroup](https://discuss.elastic.co/u/JypraGroup)\ **Post date:** [June 28, 2022, 5:42am UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/1 "2022-06-28T05:42:12Z")
Hi all, I have Elastic agent installed on the endpoint and I can see the logs coming in. The policy has AWS CloudWatch integration however I am not sure what else is required to get the logs and metrics flowing from the AWS Cloudwatch into Elastic. It seems like there are few options (access keys ans IAM role) however, the permissions required is not clear. ---
**Author:** ![Guncixx](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guncixx/32/87751_2.png) [@Guncixx](https://discuss.elastic.co/u/Guncixx)\ **Post date:** [June 28, 2022, 6:19am UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/2 "2022-06-28T06:19:51Z")
Required permissions are described in the integration documentation - ```auto #### AWS Permissions Specific AWS permissions are required for the IAM user to make specific AWS API calls. In order to enable AWS integration to collect metrics and logs from all supported service, please make sure these permissions are given: * ec2:DescribeInstances * ec2:DescribeRegions * cloudwatch:GetMetricData * cloudwatch:ListMetrics * iam:ListAccountAliases * rds:DescribeDBInstances * rds:ListTagsForResource * s3:GetObject * sns:ListTopics * sqs:ChangeMessageVisibility * sqs:DeleteMessage * sqs:ListQueues * sqs:ReceiveMessage * sts:AssumeRole * sts:GetCallerIdentity * tag:GetResources ``` ---
**Author:** ![zmoog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zmoog/32/103218_2.png) [@zmoog](https://discuss.elastic.co/u/zmoog)\ **Post date:** [June 28, 2022, 7:24am UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/3 "2022-06-28T07:24:12Z")
Hey @JypraGroup, welcome to the Elastic community! As @Guncixx already suggested, the best place to start is probably the integrations docs: > **[AWS | Elastic Documentation](https://docs.elastic.co/integrations/aws)** > > Collect logs and metrics from Amazon Web Services with Elastic Agent. The docs team recently updated and improved this page, so if you have visited it in the past, this is a perfect moment to reread it. The doc describes all your authentication options, like using the access key directly, IAM roles, and others. To learn more, check out the [AWS Credentials](https://docs.elastic.co/integrations/aws#aws-credentials) section. The most up-to-date list of permissions required is available in the [AWS Permissions](https://docs.elastic.co/integrations/aws#aws-permissions) section. ---
**Author:** ![JypraGroup](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JypraGroup](https://discuss.elastic.co/u/JypraGroup)\ **Post date:** [June 28, 2022, 9:26am UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/4 "2022-06-28T09:26:47Z")
Thanks for the warm welcome! With the access key option, it’s unclear what permission needs to be assigned to the user for the integration to work. The arn role option has the sts:assume role assigned, AWS recommends to assign minimum permissions however, the minimum required permissions are not listed in the doc. Thanks, ---
**Author:** ![zmoog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zmoog/32/103218_2.png) [@zmoog](https://discuss.elastic.co/u/zmoog)\ **Post date:** [June 28, 2022, 3:25pm UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/5 "2022-06-28T15:25:47Z")
IMO we have two problems to solve here: 1. Which authentication method to use; 2. What is the minimum set of permission required for the services we want to use. As a general rule, I suggest starting small with something simple to make it work and then iterating until we reach an optimal solution. For problem 1: the most straightforward option is to use the access and secret keys. If you already have an IAM user, create a new IAM policy with all the [permissions listed](https://docs.elastic.co/integrations/aws#aws-permissions) in the docs and attach the policy to the IAM user. Now, by using the access key and secret key for the IAM user in the integration settings, you will be able to use all supported services. For problem 2: unfortunately, the docs do not list the required permissions for each service. Let me know which integration/service you want to use, and I'll help you find the minimal permission list for your use case. We can use this work to update the docs for our future selves and other users. ---
**Author:** ![JypraGroup](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JypraGroup](https://discuss.elastic.co/u/JypraGroup)\ **Post date:** [June 29, 2022, 12:46am UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/6 "2022-06-29T00:46:24Z")
We will use the access key method. We would need minimum permissions required for the integration to work which we have assign during user creation. ![permissions](https://us1.discourse-cdn.com/elastic/original/3X/3/2/327b2381651bacbfb300a9544e61519b16ace7c0.png) ---
**Author:** ![zmoog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zmoog/32/103218_2.png) [@zmoog](https://discuss.elastic.co/u/zmoog)\ **Post date:** [June 29, 2022, 7:32am UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/7 "2022-06-29T07:32:04Z")
Great! Which integration(s) are you planning to use? Screenshots from the integration settings are okay, so I know which one you plan to use, and I can build a tailored permission list. ---
**Author:** ![JypraGroup](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JypraGroup](https://discuss.elastic.co/u/JypraGroup)\ **Post date:** [June 29, 2022, 10:35am UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/8 "2022-06-29T10:35:35Z")
Hi, I am planning to integrate with AWS CloudWatch. Regards, ---
**Author:** ![zmoog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zmoog/32/103218_2.png) [@zmoog](https://discuss.elastic.co/u/zmoog)\ **Post date:** [June 30, 2022, 1:49pm UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/9 "2022-06-30T13:49:47Z")
Hey @JypraGroup, if you're going to use CloudWatch metrics and logs. here's the IAM policy tailored to support both CloudWatch Metrics and Logs: ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "cloudwatch:GetMetricData", "cloudwatch:ListMetrics", "iam:ListAccountAliases", "ec2:DescribeRegions", "tag:GetResources", "logs:FilterLogEvents", "logs:DescribeLogGroups", "sts:GetCallerIdentity" ], "Resource": "*" } ] } ``` I forgot to mention that Filebeat and Metricbeat \[1\] documentation lists the permissions required for each module (and then integration): - [AWS CloudWatch Metricset \> AWS Permissions](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-aws-cloudwatch.html#_aws_permissions_2) - [AWS CloudWatch Input \> AWS Permissions](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-cloudwatch.html#_aws_permissions) * * * 1. The current version of the Elastic Agent orchestrates both Metricbeat and Filebeat behind the scenes to get its job done. This is going to change in future releases. ---
**Author:** ![JypraGroup](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JypraGroup](https://discuss.elastic.co/u/JypraGroup)\ **Post date:** [July 6, 2022, 6:24am UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/10 "2022-07-06T06:24:53Z")
Hi, Thanks for the details. What permissions needs to be assigned to the user who’s access and secret keys will be used within the configuration? ---
**Author:** ![JypraGroup](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JypraGroup](https://discuss.elastic.co/u/JypraGroup)\ **Post date:** [July 11, 2022, 11:39pm UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/11 "2022-07-11T23:39:30Z")
Hi Zmoog, If we have to use the access key and ARN role method, we need to associate permissions with the user when generating the keys. What those permissions will be? The ARN role that we will be using, I believe the above permissions needs to be associated with that role, is that correct? ---
**Author:** ![JypraGroup](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JypraGroup](https://discuss.elastic.co/u/JypraGroup)\ **Post date:** [July 14, 2022, 12:57am UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/12 "2022-07-14T00:57:38Z")
Hi Zmoog, the permission set are different in different documents [AWS | Elastic Documentation](https://docs.elastic.co/integrations/aws#aws-permissions) and [AWS cloudwatch metricset | Metricbeat Reference [8.3] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-aws-cloudwatch.html#_aws_permissions_2) and we have tried both and combined and still we can't see CloudWatch logs in Elastic. ---
**Author:** ![JypraGroup](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JypraGroup](https://discuss.elastic.co/u/JypraGroup)\ **Post date:** [July 19, 2022, 6:14am UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/13 "2022-07-19T06:14:48Z")
There is known bug related to CloudWatch integration [[AWS] CloudWatch logs integration fails with custom namespace and dataset · Issue #3112 · elastic/integrations · GitHub](https://github.com/elastic/integrations/issues/3112) ---
**Author:** ![zmoog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zmoog/32/103218_2.png) [@zmoog](https://discuss.elastic.co/u/zmoog)\ **Post date:** [July 20, 2022, 9:38pm UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/14 "2022-07-20T21:38:02Z")
> [@JypraGroup](#): > > There is known bug related to CloudWatch integration @JypraGroup are you using a custom namespace and dataset, as described in the issue? ---
**Author:** ![zmoog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zmoog/32/103218_2.png) [@zmoog](https://discuss.elastic.co/u/zmoog)\ **Post date:** [July 20, 2022, 9:41pm UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/15 "2022-07-20T21:41:10Z")
> [@JypraGroup](#): > > Hi Zmoog, the permission set are different in different documents [AWS | Elastic Documentation](https://docs.elastic.co/integrations/aws#aws-permissions) and [AWS cloudwatch metricset | Metricbeat Reference [8.3] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-aws-cloudwatch.html#_aws_permissions_2) and we have tried both and combined, and still we can't see CloudWatch logs in Elastic. Okay, let's see if we can understand what's going on with your setup. Can you share the logs from the Agent and Filebeat running behind the scenes? ---
**Author:** ![zmoog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zmoog/32/103218_2.png) [@zmoog](https://discuss.elastic.co/u/zmoog)\ **Post date:** [July 20, 2022, 9:45pm UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/16 "2022-07-20T21:45:54Z")
> [@JypraGroup](#): > > Hi Zmoog, If we have to use the access key and ARN role method, we need to associate permissions with the user when generating the keys. What those permissions will be? The ARN role that we will be using, I believe the above permissions needs to be associated with that role, is that correct? If you want to use an IAM role with the actual permissions, the IAM user with the access/secret key only needs the permissions to assume the role. Let me know if you are familiar with this process, otherwise, I can find an example to use as a starting point for your configuration. ---
**Author:** ![JypraGroup](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JypraGroup](https://discuss.elastic.co/u/JypraGroup)\ **Post date:** [July 20, 2022, 10:27pm UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/17 "2022-07-20T22:27:59Z")
We were using Custom namespace but switched to default as per the advice. ---
**Author:** ![JypraGroup](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JypraGroup](https://discuss.elastic.co/u/JypraGroup)\ **Post date:** [July 20, 2022, 10:30pm UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/18 "2022-07-20T22:30:47Z")
Example will be good, thanks. ---
**Author:** ![JypraGroup](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JypraGroup](https://discuss.elastic.co/u/JypraGroup)\ **Post date:** [July 20, 2022, 11:01pm UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/19 "2022-07-20T23:01:13Z")
We are getting the below error after using the default namespace. HI i am still getting the error could you please check i have created the integration in default space and configured but i still see the same error 18:19:24.155 elastic\_agent.filebeat [elastic\_agent.filebeat][warn] Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.July, 13, 19, 26, 33, 0, time.UTC), Meta:{"\_id":"36968846118257707368218283182713149301655432061887250432","raw\_index":"logs-generic-default"}, Fields:{"agent":{"ephemeral\_id":"f623e4f9-2bec-4b49-b117-2e59c66f399f","id":"423d1d45-ca8b-422a-81b5-ed12efccc3ba","name":"ip-172-31-18-17.ec2.internal","type":"filebeat","version":"8.1.2"},"awscloudwatch":{"ingestion\_time":"2022-07-13T19:26:39.000Z","log\_group":"/var/log/httpd/access\_log","log\_stream":"i-01d3e0069c2f827e9"},"cloud":{"provider":"aws","region":"us-east-1"},"data\_stream":{"dataset":"generic","namespace":"default","type":"logs"},"ecs":{"version":"8.0.0"},"elastic\_agent":{"id":"423d1d45-ca8b-422a-81b5-ed12efccc3ba","snapshot":false,"version":"8.1.2"},"event":{"dataset":"generic","id":"36968846118257707368218283182713149301655432061887250432","ingested":"2022-07-19T12:49:22.875Z"},"input":{"type":"aws-cloudwatch"},"log.file.path":"/var/log/httpd/access\_log/i-01d3e0069c2f827e9","message":"127.0.0.1 - - [13/Jul/2022:19:26:33 +0000] "GET /phpinfo.php HTTP/1.1" 200 95009 "-" "curl/7.79.1"","tags":["forwarded","aws-cloudwatch-logs"]}, Private:(\*aws.EventACKTracker)(0xc000b614d0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {"type":"security\_exception","reason":"action [indices:admin/auto\_create] is unauthorized for API key id [v5R8FoIBh-MY92CQc0UC] of user [elastic/fleet-server] on indices [logs-generic-default], this action is granted by the index privileges [auto\_configure,create\_index,manage,all]"}, dropping event! 18:19:24.155 elastic\_agent.filebeat [elastic\_agent.filebeat][warn] Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.July, 13, 19, 26, 48, 0, time.UTC), Meta:{"\_id":"36968846452858088326971752816521732839679088614004883456","raw\_index":"logs-generic-default"}, Fields:{"agent":{"ephemeral\_id":"f623e4f9-2bec-4b49-b117-2e59c66f399f","id":"423d1d45-ca8b-422a-81b5-ed12efccc3ba","name":"ip-172-31-18-17.ec2.internal","type":"filebeat","version":"8.1.2"},"awscloudwatch":{"ingestion\_time":"2022-07-13T19:26:54.000Z","log\_group":"/var/log/httpd/access\_log","log\_stream":"i-01d3e0069c2f827e9"},"cloud":{"provider":"aws","region":"us-east-1"},"data\_stream":{"dataset":"generic","namespace":"default","type":"logs"},"ecs":{"version":"8.0.0"},"elastic\_agent":{"id":"423d1d45-ca8b-422a-81b5-ed12efccc3ba","snapshot":false,"version":"8.1.2"},"event":{"dataset":"generic","id":"36968846452858088326971752816521732839679088614004883456","ingested":"2022-07-19T12:49:22.875Z"},"input":{"type":"aws-cloudwatch"},"log.file.path":"/var/log/httpd/access\_log/i-01d3e0069c2f827e9","message":"127.0.0.1 - - [13/Jul/2022:19:26:48 +0000] "GET /phpinfo.php HTTP/1.1" 200 95009 "-" "curl/7.79.1"","tags":["forwarded","aws-cloudwatch-logs"]}, Private:(\*aws.EventACKTracker)(0xc000b614d0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {"type":"security\_exception","reason":"action [indices:admin/auto\_create] is unauthorized for API key id [v5R8FoIBh-MY92CQc0UC] of user [elastic/fleet-server] on indices [logs-generic-default], this action is granted by the index privileges [auto\_configure,create\_index,mana ---
**Author:** ![JypraGroup](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JypraGroup](https://discuss.elastic.co/u/JypraGroup)\ **Post date:** [July 21, 2022, 12:03am UTC](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318/20 "2022-07-21T00:03:31Z")
The support staff on other case responded that API key doesn't have sufficient Priv. What will be your advice on what it should be set to? POST /\_security/api\_key { "name": "my-api-key", "expiration": "1d", "role\_descriptors": { "role-a": { "cluster":["monitor"], "index": [ { "names": ["logs-generic-default"], "privileges": ["manage"] } ] } } } [Next page](https://discuss.elastic.co/t/aws-cloudwatch-integration-with-elastic-using-elastic-agent/308318.md?page=2)