Elastic Serverless Forwarder - Millions of API calls causing excess costs

Elastic Stack Elasticsearch
1

The Elastic Serverless Forwarder AWS Lambda function accesses two credential values: ESF-CLOUD-ID, ESF-API-KEY
The Lambda function does not cache the credentials. It retrieves them on every invocation of the Lambda function.
For security, the credentials must be stored in a secure location. It can retrieve them from AWS Secrets Manager or AWS SSM parameter store. Both cost the exact same per every 10,000 API calls ($0.05 USD).
This is costing the company hundreds of dollars per month, purely for the API calls to the single Secrets Manager secret that contains both of the credentials. (The lower figure for one department being at least 18 million API calls per month).

The Elastic Serverless Forwarder AWS Lambda function does not support retrieving secrets through the AWS Parameters and Secrets Lambda Extension (localhost:2773), which caches values for up to 5 minutes.
Ref: Use Secrets Manager secrets in Lambda functions - AWS Lambda

Nor was it correctly coded to have the variables associated with credentials defined at module scope, so that they will persist for the lifetime of the Lambda execution environment, (being across warm invocations; but recreated on a cold start).

Will Elastic Company update the Elastic Serverless Forwarder AWS Lambda function to permit a means of caching credentials in order to save its customers money, (that is paid to AWS, not Elastic)?

2

Using the paid Elastic Support method, (since no replies have been received here, indicating the relative value of this forum), I paste in the following excerpt from an AI staff member:

"

You could try leveraging the EDOT Cloud Forwarder for AWS, which is a separate, newer Lambda-based forwarder, and that accepts the API key as a CloudFormation deployment parameter (a Lambda environment variable) rather than fetching it from Secrets Manager at invocation time, which should remove this cost category entirely. It supports CloudWatch Log Group sources for VPC Flow Logs and CloudTrail Logs today. If your deployments are forwarding those specific log types, it may be worth evaluating as a migration path. That said, it requires an Elastic Cloud Managed OTLP endpoint, and each CloudFormation stack subscribes to a single Log Group, so a deployment-per-log-group model is required.

What we are doing on the product side
We've raised an Enhancement Request (ref: #28850), asking to add configurable in-memory secret caching so that warm Lambda invocations reuse the fetched value within a TTL rather than re-calling GetSecretValue. This is the "technically correct" fix, and the related report at issue #749 confirms others have observed the same behavior. You can quote the Enhancement Request reference #28850 in any of your future exchanges with the Support team, as the Support team does not proactively provide updates on Enhancement Requests.

"

3

A belated welcome to the forum @Watsong

I understand your frustration. But this is a bit un-necessary. Elastic Cloud is a commercial product, and you have access to Elastic’s official support channels. This is a user to user community, its not part of Elastic's paid offerings.

I read your query at the time, didn't know how to answer it, so didn't. Sorry I didn't meet your SLA, I'll post "no idea" next time, so that you get a response!

That made me smile, because there are quite a few AI forum members here too. Our's hallucinate for free :slight_smile:

Going forward ...

... isn’t always the best way to encourage volunteers to spend their time helping, IMHO. YMMV.

I understand Cloud support pointed you in a promising direction, so good luck with that and your project. If someone else knows an approach/trick they might still chime in.