Cannot send error to Elastic APM from AWS lambda

[JasonREC]

Kibana version: Latest Cloud version

Elasticsearch version: Latest Cloud version

APM Server version: Latest Cloud version

APM Agent language and version:NodeJS

I followd exactly this page to do the set up (Monitoring AWS Lambda Node.js Functions | APM Node.js Agent Reference [3.x] | Elastic) setup for my app.

I use SAM tool to setup the template for cloudformation, everything is going well. Elastic APM connection is sucssful.

My app flow is simply a restful api, a request comes in then run some business logic and before I send back the response, I call a AWS SNS service since I need to run some extra background task, and the sns will send some payload to my another lambda function to proceed other api call. I used try catch block with axios there. if for example, api call failed such as token expire, error will pass to catch block, and inside catch block I simply have:

apm.setCustomContext(errorDetails)
apm.captureError(error)

errorDetails is my custom object

then after run the above 2 lines I just throw an error.

But when i check my cloud APM, I see the SNS tranaction, and it also record the api endpoint I call from my lambda, but there is no error showing at all even I call apm.captureError() since I deliberately make my token expired.

Could anyone help?

Thanks

[trentm (Trent Mick)]

Hi @JasonREC. I can try to help.

Two things would help me try to understand what is happening:

1. Are you able to get a trace log of this happening and send that to me?

To enable trace logging, you'd need to set the ELASTIC_APM_LOG_LEVEL=trace environment variable on your Lambda Function, and then re-invoke the function with the error condition (an expired token).

Then get that log output from CloudWatch Logs. In the AWS console page for your Lambda function, there is a "View CloudWatch Logs" link in the "Monitor" tab:

Then select a log group for your Lambda Function, and then "Actions > Download search results (CSV)":

If you don't want to post your log publicly, you could email them directly to me at trent.mick@elastic.co.

2. Showing some of your actual code, or sample code with the same structure would be helpful for me to see if something pops out that might be causing an issue. Or even to allow me to reproduce the issue.

[JasonREC]

Hi trentm,
Thanks for the reply.

The log is like the following

{
"log.level": "debug",
"@timestamp": "2023-03-31T09:40:53.327Z",
"log.origin": {
"file.name": "app/run.go",
"file.line": 95
},
"message": "Waiting for background data send to end",
"ecs.version": "1.6.0"
}

{
"log.level": "debug",
"@timestamp": "2023-03-31T09:40:53.327Z",
"log.origin": {
"file.name": "logsapi/event.go",
"file.line": 137
},
"message": "Current invocation over. Interrupting logs processing goroutine",
"ecs.version": "1.6.0"
}

{
"log.level": "trace",
"@timestamp": "2023-03-31T09:40:53.347Z",
"log": {
"logger": "elastic-apm-node"
},
"event.module": "apmclient",
"durationMs": 39.787120000459254,
"ecs": {
"version": "1.6.0"
},
"message": "signaled lambda invocation done"
}

{
"log.level": "trace",
"@timestamp": "2023-03-31T09:40:53.367Z",
"log": {
"logger": "elastic-apm-node"
},
"ecs": {
"version": "1.6.0"
},
"message": "lambda: wrapper end"
}

{
"log.level": "warn",
"@timestamp": "2023-03-31T09:40:53.374Z",
"log.origin": {
"file.name": "apmproxy/apmserver.go",
"file.line": 208
},
"message": "client error: response status code: 400",
"ecs.version": "1.6.0"
}

I have a module file like below, when I have an error pass to the catch block, I call the function inside the following apm.ts file.
everytime request enter my lambda, I call apm.isstart() to see if agent is ready, if not I will call apm.start(options)

import apm from 'elastic-apm-node'
export const trackError = async <T>(error: any, errorDetails: any): Promise<T>=>{
 try {
        let apmStart = apm.isStarted()
       console.log(apmStart) //this line return true when I look at cloud watch

        apm.setCustomContext(errorDetails)
        await new Promise<void>((resolve, reject) => {
            apm.captureError(error, (error: any) => {
                error ? reject(error) : resolve();
            })
        });
    } catch (error: any) {
        console.log(error)
    }}

In my template.yaml, I have the below 5 lines placed in my 2 lambda function enviroment vaiable field

NODE_OPTIONS: -r elastic-apm-node/start
ELASTIC_APM_LAMBDA_APM_SERVER: !Ref ELASTICURL
ELASTIC_APM_SECRET_TOKEN: !Ref ELASTICTOKEN
ELASTIC_APM_SEND_STRATEGY: background
ELASTIC_APM_LOG_LEVEL: trace

Thanks