# Elastic Agent, aws-s3-default-aws-s3-vpcflow keeps failing **URL:** \ **Category:** Elastic Agent\ **Created:** [September 24, 2023, 11:43pm UTC](https://discuss.elastic.co/t/elastic-agent-aws-s3-default-aws-s3-vpcflow-keeps-failing/343697 "2023-09-24T23:43:22Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![digital-thought](https://avatars.discourse-cdn.com/v4/letter/d/f14d63/32.png) [@digital-thought](https://discuss.elastic.co/u/digital-thought)\ **Post date:** [September 24, 2023, 11:43pm UTC](https://discuss.elastic.co/t/elastic-agent-aws-s3-default-aws-s3-vpcflow-keeps-failing/343697/1 "2023-09-24T23:43:22Z")
I have an AWS environment which ships VPC logs to a S3 bucket. I am using the AWS VPC Log Processing integration within Elastic Agent to process these logs and to monitor for new logs. It connects to the bucket successfully. When the agent starts up, you can see the CPU utilisation go high and stay high for considerable periods of time and the memory utilisation within Filebeat will also climb to a very high amount, so it is definitely doing something. I leave it for a couple of days, but nothing appears in my elasticsearch cluster. When I look at the logs for the elastic agent in question, I am seeing repeated messages such as: ```auto 00:17:22.551 elastic_agent [elastic_agent][info] Component state changed aws-s3-default (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '5056' exited with code '2' 00:17:22.558 elastic_agent [elastic_agent][info] Unit state changed aws-s3-default (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '5056' exited with code '2' 00:17:22.558 elastic_agent [elastic_agent][info] Unit state changed aws-s3-default-aws-s3-vpcflow-97ad6888-efd5-400e-8ca3-0a799ec519d6 (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '5056' exited with code '2' 00:17:23.755 elastic_agent [elastic_agent][info] Spawned new component aws-s3-default: Starting: spawned pid '4012' 00:17:23.756 elastic_agent [elastic_agent][info] Spawned new unit aws-s3-default-aws-s3-vpcflow-97ad6888-efd5-400e-8ca3-0a799ec519d6: Starting: spawned pid '4012' 00:17:23.756 elastic_agent [elastic_agent][info] Spawned new unit aws-s3-default: Starting: spawned pid '4012' 00:17:28.689 elastic_agent [elastic_agent][info] Component state changed aws-s3-default (STARTING->HEALTHY): Healthy: communicating with pid '4012' ``` When I check on the agent regularly, I can see different PID for filebeat which would confirm it is starting and restarting. I have a feeling the issue may be because the S3 bucket in question already has a considerable amount of VPC logs within it, and this is causing filebeat to potentially max out memory utilisation and then fail. I have tried increase memory but it ends the same way. I don't want to have to create a new S3 bucket. Would love any suggestions on how to get these logs and monitor for new ones for the VPC logs. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 25, 2023, 1:38am UTC](https://discuss.elastic.co/t/elastic-agent-aws-s3-default-aws-s3-vpcflow-keeps-failing/343697/2 "2023-09-25T01:38:26Z")
> [@digital-thought](#): > > I am using the AWS VPC Log Processing integration within Elastic Agent to process these logs and to monitor for new logs. > > It connects to the bucket successfully. Are you using it combined with SQS or directly pointing to the bucket? ---
**Author:** ![digital-thought](https://avatars.discourse-cdn.com/v4/letter/d/f14d63/32.png) [@digital-thought](https://discuss.elastic.co/u/digital-thought)\ **Post date:** [September 25, 2023, 3:15am UTC](https://discuss.elastic.co/t/elastic-agent-aws-s3-default-aws-s3-vpcflow-keeps-failing/343697/3 "2023-09-25T03:15:17Z")
Directly pointing to the bucket - no SQS ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 25, 2023, 3:39am UTC](https://discuss.elastic.co/t/elastic-agent-aws-s3-default-aws-s3-vpcflow-keeps-failing/343697/4 "2023-09-25T03:39:05Z")
Yeah, without the SQS the input will need to get a list of all the objects in the bucket and depending on the number of objects this can be really expensive. For old logs I would suggest that you set prefix by year and month until you process everything, but for new logs the best option is to use SQS. ---
**Author:** ![digital-thought](https://avatars.discourse-cdn.com/v4/letter/d/f14d63/32.png) [@digital-thought](https://discuss.elastic.co/u/digital-thought)\ **Post date:** [September 25, 2023, 3:40am UTC](https://discuss.elastic.co/t/elastic-agent-aws-s3-default-aws-s3-vpcflow-keeps-failing/343697/5 "2023-09-25T03:40:51Z")
Thanks for that - looks like the way I will go. ---
**Author:** ![digital-thought](https://avatars.discourse-cdn.com/v4/letter/d/f14d63/32.png) [@digital-thought](https://discuss.elastic.co/u/digital-thought)\ **Post date:** [September 25, 2023, 4:51am UTC](https://discuss.elastic.co/t/elastic-agent-aws-s3-default-aws-s3-vpcflow-keeps-failing/343697/6 "2023-09-25T04:51:41Z")
It is a shame the prefix does not support wildcards. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 23, 2023, 4:51am UTC](https://discuss.elastic.co/t/elastic-agent-aws-s3-default-aws-s3-vpcflow-keeps-failing/343697/7 "2023-10-23T04:51:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.