When a nodejs lambda is a target for a load balancer the traceparent/elastic-apm-headers are propagated but a new transaction is created even when the traceparent/elastic-apm-traceparent/tracestate headers are available on the request.
Looking at the code in elastic/apm-agent-nodejs this will only happen for requests done via api gateway (definition: event.requestContext && event.requestContext.requestId).
Is there any other way of getting the trace to propagate?
Thanks for reporting this @mcmikeyy -- it sounds like you're saying that an API Gateway Request made to a Node.js instrumented lambda function will always start a new distributed trace instead of continuing an existing trace. (that is -- a new transaction is always created for every new HTTP request, but transactions are linked together via a trace)
If that's the case this sounds like a bug -- we've opened an issue for this.
If you have a moment, we have a few additional questions. If you could answer as many of these as possible it will help us track down the root cause of this issue.
Do you happen to have a set of steps that can reproduce this behavior?
If not, do you have an example of the headers that are sent to the API Gateway?
Also, do you know if these initial trace headers generated by the Node.js Agent, an OpenTelemetry Agent, or some other agent/system?
I think my post was not clear enough. Sorry for the confusion I am not using an api gateway but application load balancer.
So fargate (nodejs) --> application load balancer --> lambda.
The fargate nodejs initiates the tracing, it makes a call to endpoint which resolves to an alb target which is a lambda. In the request to the lambda i can see the correct traceparent headers. In the Lambda the incoming json object is ALBEvent.
The ALBEvent contains the headers which have the traceparent within it and the requestContext which does not have a requestId unlike an api gateway which will have.
If i take this ALBEvent json and run it in the lambda test function and add a event.requestContext.requestId the service map will show an association between the two apps as the nodejs agent code assumes the request came from an api gateway request.
Initial trace headers are generated by the elastic-apm-node library package included in the fargate nodejs app.
Ah, got it, that makes sense. And the misunderstanding was a team effort I didn't realize you meant Amazon's Load Balance service -- I thought you were using the term generically.
Second -- as a work around until the above lands -- I haven't tried it but my first instinct would be to start and end a new transaction yourself in the Lambda function using the API and use the extracted headers. You can use the documented childOf option as the traceparent, and while it's not yet a formalized part of the public api, an option of tracestate should be sufficient to pass on the tracestate.
If you give option 2 a try let us know how it goes.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
I didn't realize you meant Amazon's Load Balance service -- I thought you were using the term generically.