Hi @trentm
This might be a little long message!
I am getting very flaky behavior with APM for my serverless node js application which created more confusion in understanding of its working. Will try to mention one by one point here with my observations.
-
no active transaction found - cannot build new span - For the same code deployment hardly couple of times APM was able to create transaction and rest it gave us this debug message. When I compared both logs (Created transactions vs no active transaction) I found out that there is only one change in log which is "message": "shimming handler@14.21.3 module", If I get this module in log then it will create transaction otherwise it will not. I have tried to trigger lambda from SQS message also I tried sending "Test Event" from AWS lambda UI but for both only couple of times it worked and rest it didn't work (No Transactions created and was not able to see lambda listed in APM)
-
APM Agent configuration coming in logs - I have followed exact documentation and also referred sample but in my case for below configuration its not picking up serverUrl properly.
provider:
stackTags:
Department: product
Purpose: production
Environment: ${opt:stage,'local'}
name: aws
stage: ${opt:stage,'local'}
runtime: nodejs14.x
region: ap-south-1
lambdaHashingVersion: 20201221
memorySize: 512
iamRoleStatements:
- Effect: Allow
Action:
- lambda:InvokeFunction
Resource: arn:aws:lambda:*:*:function:${self:custom.prefix}-worker
- Effect: Allow
Action:
- sqs:DeleteMessage
- sqs:ReceiveMessage
Resource: arn:aws:sqs:*:*:${self:custom.prefix}-queue
layers:
- arn:aws:lambda:ap-south-1:<account_id>:layer:elastic-apm-extension-ver-1-4-0-x86_64:1
- arn:aws:lambda:ap-south-1:<account_id>:layer:elastic-apm-node-ver-3-47-0:1
environment:
NODE_OPTIONS: -r elastic-apm-node/start
ELASTIC_APM_LAMBDA_APM_SERVER: <cloud_server_url>
ELASTIC_APM_SECRET_TOKEN: <secret_token>
ELASTIC_APM_SEND_STRATEGY: background
ELASTIC_APM_LOG_LEVEL: debug
ELASTIC_APM_ENVIRONMENT: qa
This is printed configuration in logs :
{
"log.level": "info",
"@timestamp": "2023-06-28T10:45:55.693Z",
"log": {
"logger": "elastic-apm-node"
},
"agentVersion": "3.47.0",
"env": {
"pid": 17,
"proctitle": "/var/lang/bin/node",
"os": "linux 4.14.255-311-248.529.amzn2.x86_64",
"arch": "x64",
"host": "169.254.217.193",
"timezone": "UTC+00",
"runtime": "Node.js v14.21.3"
},
"config": {
"serviceName": {
"source": "default",
"value": "qa-daily-report-worker",
"commonName": "service_name"
},
"serviceVersion": {
"source": "default",
"value": "47",
"commonName": "service_version"
},
"serverUrl": {
"source": "default",
"value": "http://127.0.0.1:8200/",
"commonName": "server_url"
},
"logLevel": {
"source": "environment",
"value": "debug",
"commonName": "log_level"
},
"environment": {
"source": "environment",
"value": "qa"
},
"secretToken": {
"source": "environment",
"value": "[REDACTED]",
"commonName": "secret_token"
}
},
"activationMethod": "aws-lambda-layer",
"ecs": {
"version": "1.6.0"
},
"message": "Elastic APM Node.js Agent v3.47.0"
}
if you see serverUrl here is (http://127.0.0.1:8200/) even though I added elastic cloud APM Url. If I add additional environment variable ELASTIC_APM_SERVER_URL with my APM url then it shows up correctly in log. So do I need to add both environment variables (ELASTIC_APM_SERVER_URL ,ELASTIC_APM_LAMBDA_APM_SERVER) to setup correct APM?
If I remove ELASTIC_APM_LAMBDA_APM_SERVER and just keep ELASTIC_APM_SERVER_URL then it gives error that Server Url can not be empty.
- As per your last suggestion I want to add customContext to capture events for each lambda invocations, but since we have added elastic-node-apm dependency as layer in serverless.yml I cant use that dependency directly in my handler code, so to do that I need to add
"elastic-apm-node": "^3.47.0" in my package.json is it correct way to configure? Also since my serverless take care of starting APM agent I think I just need to retrieve APM agent and use it to set customContext as below.
export const dailyReport = async (event, context, callback) => {
try {
const apm = require('elastic-apm-node');
apm.setCustomContext(event);
// More code ...
return {
statusCode: 200,
body: 'ok'
};
} catch (err) {
logger.error({message: err.message, stack: err.stack});
callback(err);
}
Since my APM is not creating transactions most of the times I am not able test my above code. So just confirming if its the right way to set context.
Let me know if I need to provide more logs 
Thank you for your patience for reading such a long message!
