How to send error message to Elastic APM and show the error in Kibana tab

[JasonREC]

I followed the set up of this page

Monitoring AWS Lambda Node.js Functions | APM Node.js Agent Reference [4.x] |...

I am using the template.yml to set up my lambda function and Elastic APM

...
Resources:
  yourLambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      ...
      Environment:
          Variables:
            NODE_OPTIONS: -r elastic-apm-node/start
            ELASTIC_APM_LAMBDA_APM_SERVER: <YOUR-APM-SERVER-URL>
            ELASTIC_APM_SECRET_TOKEN: <YOUR-APM-SECRET-TOKEN>
            ELASTIC_APM_SEND_STRATEGY: background

According to the offical webpage it said,

By default, the Node.js agent will watch for uncaught exceptions and send them to Elastic APM automatically. But in most cases, errors are not thrown but returned via a callback, caught by a promise, or simply manually created. Those errors will not automatically be sent to Elastic APM. To manually send an error to Elastic APM, simply call apm.captureError() with the error:

var err = new Error('Ups, something broke!') 

apm.captureError(err)

I am wonder if I have to npm install the 'elastic-apm-node' and import it into my code so as to manually send an error to Elastic APM, like apm.captureError(err)

import apm from 'elastic-apm-node'
 apm.start({
 
            serviceName: serviecName,

            secretToken: secretToken,

            serverUrl: serviceUrl,
        })
apm.captureError(err)

But I have already stated that NODE_OPTIONS: -r elastic-apm-node/start in my template.yml, wouldn't it be started the elastic apm agent twice!?

Could anyone help?
Thanks

[trentm (Trent Mick)]

Hi @JasonREC. Thanks for the question.

You are correct that using apm.start() again would start the APM agent twice, which would break.

In your Lambda code you should be able to import the APM module and use it without .start()ing it again.

import apm from 'elastic-apm-node'
apm.captureError(err)

That import should return the already-started APM agent singleton object.
(Note that sometime there can be a gotcha here if bundling is involved.)

[JasonREC]

Hi @trentm ,Thanks for the reply

I found out this would show an undefined agent somhow if I dont use agent start method twice.

Below is my lambda function entry point, I've already stated : NODE_OPTIONS
-r elastic-apm-node/start in my tamplate.yml, and I called the apm.isStated method, but it return undefined

import apm from 'elastic-apm-node'
import { Handler, SNSEvent } from "aws-lambda";
import Model from './Model';
import timer from './lib/timer';
import { run as runBackgroundTask } from './BackgroundTask';

export const handler: Handler<SNSEvent, string | void> = async (event) => {
    console.log(apm.isStarted(), 'check start or not ') // here prints undefined
    timer.reset();

    console.log('enter bg lambda function');
    const result = await Promise.all(event.Records
        .map(record => JSON.parse(record.Sns.Message) as Model.BackgroundTask)
        .map(runBackgroundTask));

    console.log('All task done', result);
    return "Success";

};

[trentm (Trent Mick)]

@JasonREC Hrm. That undefined result means that your import apm from 'elastic-apm-node' is creating a different APM Agent class instance from the one created by NODE_OPTIONS=.... They are meant to be the same singleton.

This is TypeScript code. Are you able to post the built JavaScript code that you are deploying? Perhaps it is emitting ESM code (using import ...) rather than CommonJS code (using require). If so, then you are liking hitting the APM agent's limitations with ES modules. See Supported technologies | APM Node.js Agent Reference [3.x] | Elastic for some details.

[JasonREC]

Hi @trentm,

Appreciate for your quick reply.

I see your point. Currently, I am using template.yml MetaData property to do the build

Building Node.js Lambda functions with esbuild - AWS Serverless Application...

And, I just add one more line in template.yml to specify that I want it to help me compile Typescript to Javascript in Common JS. I think this may satisify that the elastic document points out

The Elastic APM Node.js agent does not yet support automatically instrumenting ECMAScript module imports (ESM) , i.e. modules that are loaded via import ... statements. It currently only instruments CommonJS module imports, i.e. modules loaded via require(...) .

Metadata: # Manage esbuild properties
BuildMethod: esbuild
BuildProperties:
Format: cjs <------ I add this line
Minify: true
Target: "es2020"
Sourcemap: true
EntryPoints:
- lambdaBackground.ts

But when I looked at the cloudwatch log, it still said apm agent is undfined when the request enter the lambda function.

And, if I run apm.captureError(new Error('test')), it will say agent has not started

I wish I could post the built javascript code, but they all got minified, and it is almost unreadble.

[trentm (Trent Mick)]

Oh, esbuild and bundling/minifying are involved. That will be an issue. See Starting the agent | APM Node.js Agent Reference [4.x] | Elastic (and the "esbuild" section further down) for some details.

I don't have an answer that I know will work. I've opened get working out of the box with AWS Lambda using esbuild: Agent singleton, esbuild format config, etc. · Issue #3244 · elastic/apm-agent-nodejs · GitHub to look into this when I am able.

A couple things you could try now as workarounds are points 2 and 3 on that issue:

2. Another workaround might be to specify that the elastic-apm-node module be in the esbuild externals -- as documented at Starting the agent | APM Node.js Agent Reference [4.x] | Elastic. Can this config be passed through via the SAM configuration? If so, we should document this example.
3. Another possible workaround might be to not have elastic-apm-node listed in dependencies, but still import it via CommonJS require. That might result in getting the elastic-apm-node module provided by the Elastic Lambda layer. This may require configuring a linter and/or esbuild to ignore the attempt to import a module that is not listed in "dependencies".

[JasonREC]

Hi, @trentm
Thanks for helping me.
I looked at the document you mentioned, it is very helpful to know so as to use elastic apm node with ESbuild, I have to mark the modules as external.

And, yes, I just noticed that in the AWS document (The intro of Building Node.js Lambda functions with esbuild that I mentioned earlier), it does have a field called External, which the usage is

Specifies the list of packages to omit from the build.

I think I would love to give it a try and see if this can fix the problem. I will let you how it goes as soon as I try it.

Thanks

[JasonREC]

Hi @trentm,

Adding external does fix the undefined agent problem, and error message now can sucessfully send to Kibana tab. Appreciate for your help.

[trentm (Trent Mick)]

@JasonREC Good to hear. Thanks for following up on this thread!