I have deployed an elastic agent with the AWS Integration in order to process VPC and CloudTrail logs.
The logs are being placed to seperate S3 buckets which in turn then trigger an event to an SQS queue - one for the VPC and one for the Cloudtrail.
I have setup the integration to process via the SQS and I have setup the integration to process uto 250 messages simultaneously.
The machine on which the agent is running has 16 cores and 60 GiB of memory available to it.
One they agent starts, you look at the SQS for both the VPC and Cloudtrail and you can see 250 messages are in flight for both, however, it then quickly drops away to only having 50 or 30 messages in flight.
The queue quickly grows and the agent never catches up and as a result log processing is WAY behind.
What can be done to improvide performance of the elastic integration for AWS so it keeps up with the logs being placed to the S3?
Event adding more agents doesnt help the situation.