APM engine not sampling all requests

[Abhiraj_Kulaar]

Hi Team,

We have an implementation of NodeJS APM agent running in our Firebase server-less functions along with custom spans in some of the transactions. After trying to compare logs on the APM console with application logs we found out that the APM agent is missing some requests entirely and are not being logged onto the server. We have failed to find any co-relation of request type/time between the missing requests. Upon further investigation I started returning the object apm.transaction in request responses and found out that the requests which fail to be logged have the following JSON.stringify value:

{"id":"REDACTED",
"trace_id":"REDACTED",
"parent_id":"REDACTED",
"name":"GET unknown route (unnamed)",
"type":"request",
"duration":null,
"timestamp":1645644378633011,
"result":"success",
"sampled":false,
"span_count":{"started":0},
"outcome":"unknown"}.

From what I observe sampled is false here eventhough i have 1.0 sampling rate.

I also suspect this maybe because of the firebase environment as it works flawlessly on local firebase emulator.

Kindly help me solve this any way.

[alanstorm]

Thanks for writing and reporting the behavior @Abhiraj_Kulaar.

If I'm reading what you said correctly -- it sounds like you have some Google Cloud Serverless Functions running in combination with your Google Firebase applications (as outlined here: Cloud Functions for Firebase  |  Firebase Documentation), and APM data is not consistently coming though.

Unfortunately, the Elastic APM Node.js agent doesn't officially support Google Cloud Serverless Functions. Due to the unique execution environment of Cloud Functions it's not always possible to get data out of the function after it finishes executing. The sampling you're seeing isn't a result of agent sampling behavior, it's a result of how cloud functions run in Google's production environment.

We'll make a note of this as a feature request for Google Cloud Functions. Also, if you happen to find a way to get it working please do share with the community.

Best wishes.

[alanstorm]

Hey @Abhiraj_Kulaar -- one additional thing: The reason you're seeing sampled:false on those transactions -- sometimes Google's infrastructure can add a Traceparent header to the request before it reaches your service, indicating that server should be sampled. You [can read a bit more about this behavior in the following thead]Traceparent header from outside Elastic · Issue #286 · elastic/apm · GitHub).