We're attempting to configure a Functionbeat that subscribes to multiple CloudWatch Log Groups across our stacks. These stacks are created dynamically through independent CI/CD pipelines, so we'd like the responsibility of establishing a subscription to the Functionbeat to lie with the stacks generating the logs, not the Functionbeat. This will allow us to have a single Functionbeat stack, with new stacks subscribing to it as they are created, and removing their subscription as they are deleted.
In theory this should be possible by adding a Subscription Filter to each stack, as below
FunctionbeatLambdaLogGroupSubscription: Properties: DestinationArn: arn:my-functionbeat-function FilterPattern: "" LogGroupName: /aws/lambda/my-log-group Type: 'AWS::Logs::SubscriptionFilter'
However, when I try to set up the subscription in this way, without specifying any
triggers in my Functionbeat.yml file, I receive the following error when my Functionbeat is executed:
Exiting: error when creating the functions, error: you need to specify at least one trigger accessing 'functionbeat.provider.aws.functions.0' (source:'functionbeat.yml')
I take from this that it is required to list out each Log Group that the Functionbeat should be triggered by. Is this just a validation step, or is there some logic within the Functionbeat that requires it to have this configuration on top of the Log Group subscription?
From this thread, (https://github.com/elastic/beats/issues/10756) it doesn't sound like its possible to wildcard these Triggers. Listing out each Log Group to subscribe to isn't feasible in our case, and I imagine in many others.
Is it possible to configure a Functionbeat without specifying these
triggers in its configuration? And if not, is this something that's under consideration for a future release? Or is there another architecture approach we should be considering for this that Functionbeat was designed for