Configure AWS Log Group Subscriptions without Triggers in Functionbeat.yml file

[PhilipMortiboy]

We're attempting to configure a Functionbeat that subscribes to multiple CloudWatch Log Groups across our stacks. These stacks are created dynamically through independent CI/CD pipelines, so we'd like the responsibility of establishing a subscription to the Functionbeat to lie with the stacks generating the logs, not the Functionbeat. This will allow us to have a single Functionbeat stack, with new stacks subscribing to it as they are created, and removing their subscription as they are deleted.

In theory this should be possible by adding a Subscription Filter to each stack, as below

FunctionbeatLambdaLogGroupSubscription:
  Properties:
    DestinationArn: arn:my-functionbeat-function
      FilterPattern: ""
      LogGroupName: /aws/lambda/my-log-group
  Type: 'AWS::Logs::SubscriptionFilter'

However, when I try to set up the subscription in this way, without specifying any triggers in my Functionbeat.yml file, I receive the following error when my Functionbeat is executed:

Exiting: error when creating the functions, error: you need to specify at least one trigger accessing 'functionbeat.provider.aws.functions.0' (source:'functionbeat.yml')

I take from this that it is required to list out each Log Group that the Functionbeat should be triggered by. Is this just a validation step, or is there some logic within the Functionbeat that requires it to have this configuration on top of the Log Group subscription?

From this thread, (https://github.com/elastic/beats/issues/10756) it doesn't sound like its possible to wildcard these Triggers. Listing out each Log Group to subscribe to isn't feasible in our case, and I imagine in many others.

Is it possible to configure a Functionbeat without specifying these triggers in its configuration? And if not, is this something that's under consideration for a future release? Or is there another architecture approach we should be considering for this that Functionbeat was designed for

Thanks

[mtojek (Marcin Tojek)]

I think you need to come back to designing the architecture. Functionbeat without triggers doesn't sound like a live code that can be executed.

Out out curiosity, why do you think that listing log groups is not feasible? Costs?

I admin I don't know your case, but did you consider creating dedicated lambdas bound together with your stacks (lambdas would be aware of Log Groups)?

[PhilipMortiboy]

Hi Marcin

Thanks for the quick reply

The issue with listing the log groups is not cost, but complexity. If you consider the architecture below, which is what we'd like to build, having a situation where the Functionbeat needs to be configured with a list of every log group across all stacks in its triggers config is not feasible at scale. We have a lot of stacks, and these are created, updated and deleted frequently. Having to update the Functionbeat's config, package and deploy it every time adds a lot of complexity to our pipelines. And its very difficult, if not impossible, to keep this list of triggers up to date in an automated fashion

We have considered creating dedicate Functionbeat Lambdas for each stack as you suggest. This is actually the approach we're going with for now. It gets around the problem of having to maintain a list of all log groups in the trigger, as the Functionbeat now only cares about the Log Groups within its stack

The issue here is that we now have multiple Functionbeats across our stacks. So if we want to update to a newer version of Functionbeat, or make global changes to the config, this now has to be done for each stack independently

What I'm not clear on is why the Functionbeat config requires a list of Log Group triggers when we already have the CloudWatch Subscription Filter in place? These Subscriptions alone are enough to trigger the Lambda function and initiate the Functionbeat. As per the AWS documentation, the log events are sent to the function as part of the payload, (https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SubscriptionFilters.html#LambdaFunctionExample). What does the Functionbeat do with the Log Group triggers defined in its config?

If we were able to set up a Functionbeat with only the CloudWatch Subscription Filter, then it would be possible to achieve the architecture outlined in my first diagram

[cluggas (Andy McCluggage)]

I think Philip's approach makes sense.

The AWS architecture has a one-way coupling from the Log Group Subscription to the Log Group Subscription Target. In other words, the Log Group Subscription Target Lambda function (which is functionbeat in this case) doesn't need need to know about the Log Group Subscriptions that target it.

Functionbeat does need to know, so this goes against the AWS architecture.

The number of Subscriptions pointing to a Target should be able to scale up without having to modify the Target. This would allow everything to scale nicely. Having a different functionbeat per Stack is not sustainable at scale.