[AWS] CloudFormation Stack Resource Limits

I am currently testing out Functionbeat for forwarding CloudWatch logs to my Logstash service. From what I have gathered, under the hood Functionbeat is generating a CloudFormation stack template for all necessary resources and then attempting to deploy that template. Unfortunately, we currently have over 100 CloudWatch log groups, and because Functionbeat is creating a SubscriptionFilter and Lambda execution permission for each, we have far surpassed the 200 resource limit for CloudFormation stacks (unfortunately one of the hard resource limits).

I would very much like to use Functionbeat in our AWS environment, and for the time being will probably be trying to break our CloudwatchLogs into logical groups and deploying a Functionbeat for each, but I would prefer to have a single deployment. Do you have any plans to either use multiple stacks or nested stacks for your deployment? Having a nested stack for CloudWatch Logs SubscriptionFilters and another one for Lambda Permissions (as well as separate nested stacks for whatever the SQS and Kinesis subscription resources are) would allow for up to roughly 200 triggers (more if you dynamically add more nested stacks when you hit the resource limit).

We might need to be rethinking how we're manage our CloudWatch Log Groups so that we have fewer of them, but I wanted to bring this issue to your attention as a potential area for enhancement in the future!

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.