Elastic Agent : AWS CloudWatch Fleet Integration Fail with RequestCanceledError

Elastic Stack Beats
1

Hi,

i'm trying to set up an AWS integration to bring Lambda log group.
I'm already using the AWS integration to have some metrics, so a don't think is a credential problem. I followed this guide AWS | Elastic Documentation and i give all the necessary permissions to my user.

This is a piece of filebeat log under:

{"log.level":"info","@timestamp":"2022-07-20T07:23:48.942Z","log.logger":"input.aws-cloudwatch.cloudwatch_poller","log.origin":{"file.name":"awscloudwatch/input.go","file.line":213},"message":"aws-cloudwatch input worker for log group: '/aws/lambda/APIShare_Cognito_Authorizers' has started","service.name":"filebeat","id":"aws-cloudwatch-aws.cloudwatch_logs-5f521d35-5207-4f0b-b5f3-33cf735f4b78","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-07-20T07:23:48.964Z","log.logger":"input.aws-cloudwatch.cloudwatch_poller","log.origin":{"file.name":"awscloudwatch/cloudwatch.go","file.line":70},"message":"getLogEventsFromCloudWatch failed with RequestCanceledError: <nil>","service.name":"filebeat","id":"aws-cloudwatch-aws.cloudwatch_logs-5f521d35-5207-4f0b-b5f3-33cf735f4b78","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-07-20T07:23:48.964Z","log.logger":"input.aws-cloudwatch.cloudwatch_poller","log.origin":{"file.name":"awscloudwatch/cloudwatch.go","file.line":72},"message":"getLogEventsFromCloudWatch failed: <nil>","service.name":"filebeat","id":"aws-cloudwatch-aws.cloudwatch_logs-5f521d35-5207-4f0b-b5f3-33cf735f4b78","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-20T07:23:48.964Z","log.logger":"input.aws-cloudwatch.cloudwatch_poller","log.origin":{"file.name":"awscloudwatch/input.go","file.line":209},"message":"aws-cloudwatch input worker for log group '/aws/lambda/APIShare_Cognito_Authorizers' has stopped.","service.name":"filebeat","id":"aws-cloudwatch-aws.cloudwatch_logs-5f521d35-5207-4f0b-b5f3-33cf735f4b78","ecs.version":"1.6.0"}
2

hello @davide.lilliu , welcome to the Elastic community

the error reported in the log is misleading: we already reported the bug and it should be fixed in the next release

to clarify: it's only the fact that the error is reported as an empty Request Canceled error the bogus behaviour

but an error indeed occurred in your setup: due to the bug mentioned above we lost the real error type and message

most likely it is a permission problem.

the actions required to fetch metrics are different from the ones required to fetch cloudwatch logs events

I can indeed see that the docs does not mention logs:FilterLogEvents, this is indeed required for fetching cloudwatch logs events.

could you try to add it to the IAM resource you use?

we will fix the documentation as well

3

Hi @Andrea_Spacca ,

thanks for the answer and for welcome.

i add the aws permission that you told me, but it doesn't work again, now we have a different message on filebeat logs.

{"log.level":"warn","@timestamp":"2022-07-22T12:34:28.385Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.April, 26, 12, 34, 45, 0, time.UTC), Meta:{\"_id\":\"36818005933506741614627722203827005425098558727599030300\",\"raw_index\":\"logs-generic-default\"}, Fields:{\"agent\":{\"ephemeral_id\":\"1b80399f-afbb-426c-b974-3ae9af85977b\",\"id\":\"440b5ec3-9985-41e6-887f-3259900d61d9\",\"name\":\"ip-172-40-5-254.eu-west-1.compute.internal\",\"type\":\"filebeat\",\"version\":\"8.3.2\"},\"aws.cloudwatch\":{\"ingestion_time\":\"2022-04-26T12:34:53.000Z\",\"log_group\":\"/aws/lambda/APIShare_Cognito_Authorizers\",\"log_stream\":\"2022/04/26/[$LATEST]af7d872c0e3a41f9bba7f8bfac7fe00a\"},\"awscloudwatch\":{\"ingestion_time\":\"2022-04-26T12:34:53.000Z\",\"log_group\":\"/aws/lambda/APIShare_Cognito_Authorizers\",\"log_stream\":\"2022/04/26/[$LATEST]af7d872c0e3a41f9bba7f8bfac7fe00a\"},\"cloud\":{\"provider\":\"aws\",\"region\":\"eu-west-1\"},\"container\":{\"id\":\"04\"},\"data_stream\":{\"dataset\":\"generic\",\"namespace\":\"default\",\"type\":\"logs\"},\"ecs\":{\"version\":\"8.0.0\"},\"elastic_agent\":{\"id\":\"440b5ec3-9985-41e6-887f-3259900d61d9\",\"snapshot\":false,\"version\":\"8.3.2\"},\"event\":{\"dataset\":\"generic\",\"id\":\"36818005933506741614627722203827005425098558727599030300\",\"ingested\":\"2022-07-22T12:33:33.054Z\"},\"input\":{\"type\":\"aws-cloudwatch\"},\"log.file.path\":\"/aws/lambda/APIShare_Cognito_Authorizers/2022/04/26/[$LATEST]af7d872c0e3a41f9bba7f8bfac7fe00a\",\"message\":\"2\\n\",\"tags\":[\"forwarded\",\"aws-cloudwatch-logs\"]}, Private:(*aws.EventACKTracker)(0xc000643560), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:admin/auto_create] is unauthorized for API key id [yjLUJYIBu4Mq7F4vLShT] of user [elastic/fleet-server] on indices [logs-generic-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]\"}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-07-22T12:34:28.386Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.April, 26, 12, 34, 45, 0, time.UTC), Meta:{\"_id\":\"36818005933506741614627722203827005425098558727599030301\",\"raw_index\":\"logs-generic-default\"}, Fields:{\"agent\":{\"ephemeral_id\":\"1b80399f-afbb-426c-b974-3ae9af85977b\",\"id\":\"440b5ec3-9985-41e6-887f-3259900d61d9\",\"name\":\"ip-172-40-5-254.eu-west-1.compute.internal\",\"type\":\"filebeat\",\"version\":\"8.3.2\"},\"aws.cloudwatch\":{\"ingestion_time\":\"2022-04-26T12:34:53.000Z\",\"log_group\":\"/aws/lambda/APIShare_Cognito_Authorizers\",\"log_stream\":\"2022/04/26/[$LATEST]af7d872c0e3a41f9bba7f8bfac7fe00a\"},\"awscloudwatch\":{\"ingestion_time\":\"2022-04-26T12:34:53.000Z\",\"log_group\":\"/aws/lambda/APIShare_Cognito_Authorizers\",\"log_stream\":\"2022/04/26/[$LATEST]af7d872c0e3a41f9bba7f8bfac7fe00a\"},\"cloud\":{\"provider\":\"aws\",\"region\":\"eu-west-1\"},\"container\":{\"id\":\"04\"},\"data_stream\":{\"dataset\":\"generic\",\"namespace\":\"default\",\"type\":\"logs\"},\"ecs\":{\"version\":\"8.0.0\"},\"elastic_agent\":{\"id\":\"440b5ec3-9985-41e6-887f-3259900d61d9\",\"snapshot\":false,\"version\":\"8.3.2\"},\"event\":{\"dataset\":\"generic\",\"id\":\"36818005933506741614627722203827005425098558727599030301\",\"ingested\":\"2022-07-22T12:33:33.054Z\"},\"input\":{\"type\":\"aws-cloudwatch\"},\"log.file.path\":\"/aws/lambda/APIShare_Cognito_Authorizers/2022/04/26/[$LATEST]af7d872c0e3a41f9bba7f8bfac7fe00a\",\"message\":\"0\\n\",\"tags\":[\"forwarded\",\"aws-cloudwatch-logs\"]}, Private:(*aws.EventACKTracker)(0xc000643560), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:admin/auto_create] is unauthorized for API key id [yjLUJYIBu4Mq7F4vLShT] of user [elastic/fleet-server] on indices [logs-generic-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]\"}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-07-22T12:34:28.386Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.April, 26, 12, 34, 45, 0, time.UTC), Meta:{\"_id\":\"36818005933506741614627722203827005425098558727599030302\",\"raw_index\":\"logs-generic-default\"}, Fields:{\"agent\":{\"ephemeral_id\":\"1b80399f-afbb-426c-b974-3ae9af85977b\",\"id\":\"440b5ec3-9985-41e6-887f-3259900d61d9\",\"name\":\"ip-172-40-5-254.eu-west-1.compute.internal\",\"type\":\"filebeat\",\"version\":\"8.3.2\"},\"aws.cloudwatch\":{\"ingestion_time\":\"2022-04-26T12:34:53.000Z\",\"log_group\":\"/aws/lambda/APIShare_Cognito_Authorizers\",\"log_stream\":\"2022/04/26/[$LATEST]af7d872c0e3a41f9bba7f8bfac7fe00a\"},\"awscloudwatch\":{\"ingestion_time\":\"2022-04-26T12:34:53.000Z\",\"log_group\":\"/aws/lambda/APIShare_Cognito_Authorizers\",\"log_stream\":\"2022/04/26/[$LATEST]af7d872c0e3a41f9bba7f8bfac7fe00a\"},\"cloud\":{\"provider\":\"aws\",\"region\":\"eu-west-1\"},\"container\":{\"id\":\"04\"},\"data_stream\":{\"dataset\":\"generic\",\"namespace\":\"default\",\"type\":\"logs\"},\"ecs\":{\"version\":\"8.0.0\"},\"elastic_agent\":{\"id\":\"440b5ec3-9985-41e6-887f-3259900d61d9\",\"snapshot\":false,\"version\":\"8.3.2\"},\"event\":{\"dataset\":\"generic\",\"id\":\"36818005933506741614627722203827005425098558727599030302\",\"ingested\":\"2022-07-22T12:33:33.054Z\"},\"input\":{\"type\":\"aws-cloudwatch\"},\"log.file.path\":\"/aws/lambda/APIShare_Cognito_Authorizers/2022/04/26/[$LATEST]af7d872c0e3a41f9bba7f8bfac7fe00a\",\"message\":\"key index 0\\n\",\"tags\":[\"forwarded\",\"aws-cloudwatch-logs\"]}, Private:(*aws.EventACKTracker)(0xc000643560), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:admin/auto_create] is unauthorized for API key id [yjLUJYIBu4Mq7F4vLShT] of user [elastic/fleet-server] on indices [logs-generic-default], this action is granted by the index privileges [auto_configure,create_index,manage,all]\"}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-22T12:36:32.443Z","log.logger":"input.aws-cloudwatch.cloudwatch_poller","log.origin":{"file.name":"awscloudwatch/input.go","file.line":213},"message":"aws-cloudwatch input worker for log group: '/aws/lambda/APIShare_Cognito_Authorizers' has started","service.name":"filebeat","id":"aws-cloudwatch-aws_logs.generic-1a07a8ba-6b51-4569-bec2-9798c98d478f","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-07-22T12:36:35.123Z","log.logger":"input.aws-cloudwatch.cloudwatch_poller","log.origin":{"file.name":"awscloudwatch/input.go","file.line":213},"message":"aws-cloudwatch input worker for log group: '/aws/lambda/APIShare_Cognito_Authorizers' has started","service.name":"filebeat","id":"aws-cloudwatch-aws_logs.generic-1a07a8ba-6b51-4569-bec2-9798c98d478f","ecs.version":"1.6.0"}
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.