APM - new transaction created when lambda requests via aws application load balancer

A lambda nodejs app has

  • apm package "elastic-apm-node": "3.38.0"
  • layer extension: layer:elastic-apm-extension-ver-1-1-0-x86_64:1
  • using elastic/kibana: 8.2.3

When a nodejs lambda is a target for a load balancer the traceparent/elastic-apm-headers are propagated but a new transaction is created even when the traceparent/elastic-apm-traceparent/tracestate headers are available on the request.

Looking at the code in elastic/apm-agent-nodejs this will only happen for requests done via api gateway (definition: event.requestContext && event.requestContext.requestId).

Is there any other way of getting the trace to propagate?

1 Like

Thanks for reporting this @mcmikeyy -- it sounds like you're saying that an API Gateway Request made to a Node.js instrumented lambda function will always start a new distributed trace instead of continuing an existing trace. (that is -- a new transaction is always created for every new HTTP request, but transactions are linked together via a trace)

If that's the case this sounds like a bug -- we've opened an issue for this.

If you have a moment, we have a few additional questions. If you could answer as many of these as possible it will help us track down the root cause of this issue.

  1. Do you happen to have a set of steps that can reproduce this behavior?

  2. If not, do you have an example of the headers that are sent to the API Gateway?

  3. Also, do you know if these initial trace headers generated by the Node.js Agent, an OpenTelemetry Agent, or some other agent/system?

I think my post was not clear enough. Sorry for the confusion I am not using an api gateway but application load balancer.

So fargate (nodejs) --> application load balancer --> lambda.

The fargate nodejs initiates the tracing, it makes a call to endpoint which resolves to an alb target which is a lambda. In the request to the lambda i can see the correct traceparent headers. In the Lambda the incoming json object is ALBEvent.

The ALBEvent contains the headers which have the traceparent within it and the requestContext which does not have a requestId unlike an api gateway which will have.

If i take this ALBEvent json and run it in the lambda test function and add a event.requestContext.requestId the service map will show an association between the two apps as the nodejs agent code assumes the request came from an api gateway request.

Initial trace headers are generated by the elastic-apm-node library package included in the fargate nodejs app.

"elastic-apm-traceparent"
"traceparent"
"tracestate"

At the moment i have deployed apps in aws where i can recreate this.

The lambda uses Starting node with -r elastic-apm-node/start

  • node14
  • apm package "elastic-apm-node": "3.38.0"
  • layer extension: layer:elastic-apm-extension-ver-1-1-0-x86_64:1

Ah, got it, that makes sense. And the misunderstanding was a team effort :slight_smile: I didn't realize you meant Amazon's Load Balance service -- I thought you were using the term generically.

First -- this is actually on our radar to get working -- the specific Node.js issue to watch is here: [META 679] Add support for ELB and Lambda URL metadata in lambda integrations · Issue #2901 · elastic/apm-agent-nodejs · GitHub, which will link to the meta issue

Second -- as a work around until the above lands -- I haven't tried it but my first instinct would be to start and end a new transaction yourself in the Lambda function using the API and use the extracted headers. You can use the documented childOf option as the traceparent, and while it's not yet a formalized part of the public api, an option of tracestate should be sufficient to pass on the tracestate.

If you give option 2 a try let us know how it goes.

Hi,

Thanks for the update. Tried option 2 and it didnt seem to generate any meaningful data. I am looking at open telemetry to see if that supports elb.