# Not able to find a way to add event.json in APM serverless Lambda transactions

**URL:** <https://discuss.elastic.co/t/not-able-to-find-a-way-to-add-event-json-in-apm-serverless-lambda-transactions/336984>\
**Category:** APM\
**Tags:** nodejs\
**Created:** [June 27, 2023, 11:58am UTC](https://discuss.elastic.co/t/not-able-to-find-a-way-to-add-event-json-in-apm-serverless-lambda-transactions/336984 "2023-06-27T11:58:20Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kesha\_Shah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kesha_shah/32/122625_2.png) [@Kesha\_Shah](https://discuss.elastic.co/u/Kesha_Shah)\
**Post date:** [June 27, 2023, 11:58am UTC](https://discuss.elastic.co/t/not-able-to-find-a-way-to-add-event-json-in-apm-serverless-lambda-transactions/336984/1 "2023-06-27T11:58:20Z")

</div>

**Kibana version** : 8.8.1

**Elasticsearch version** : 8.8.1

**APM Server version** : 8.8.1

**APM Agent language and version** : NodeJs Agent `elastic-apm-node": "^3.47.0`

**Fresh install or upgraded from other version?** Upgraded from 7.17 recently

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:

I am using AWS lambda with Nodejs and serverless - following example [here](https://www.elastic.co/guide/en/apm/agent/nodejs/current/lambda.html)

By following documentations I am able to get basic traces of my functions but I also want to add event body for my transactions at least for errors.

My lambda functions is invoked by SQS message so my transaction type comes as `messaging`.

Below is my current configuration

```auto
layers:
    - arn:aws:lambda:ap-south-1:267093732750:layer:elastic-apm-extension-ver-1-4-0-x86_64:1
    - arn:aws:lambda:ap-south-1:267093732750:layer:elastic-apm-node-ver-3-47-0:1
  environment:
    NODE_OPTIONS: -r elastic-apm-node/start
    ELASTIC_APM_LAMBDA_APM_SERVER: <server-url>
    ELASTIC_APM_SECRET_TOKEN: <token>
    ELASTIC_APM_SEND_STRATEGY: background
    ELASTIC_APM_LOG_LEVEL: info
    ELASTIC_APM_ENVIRONMENT: ${file(../../environment/${self:provider.stage}.yml):ENV}

```

Although I knew that mostly `captureBody` wont work with queue message as event, I still tried adding environment variable `ELASTIC_APM_CAPTURE_BODY: transactions` but with that my transactions stopped coming to APM with below debug message

```auto
{
    "log.level": "debug",
    "@timestamp": "2023-06-27T11:36:48.222Z",
    "log": {
        "logger": "elastic-apm-node"
    },
    "ecs": {
        "version": "1.6.0"
    },
    "message": "no active transaction found - cannot build new span"
}

```

I have checked all other configuration variables but I am not able to find any suitable config to add event.json for my lambda function.

Any help is appreciated 🙂

---

<div class="post-metadata">

**Author:** ![trentm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trentm/32/77647_2.png) [@trentm](https://discuss.elastic.co/u/trentm)\
**Post date:** [June 27, 2023, 4:25pm UTC](https://discuss.elastic.co/t/not-able-to-find-a-way-to-add-event-json-in-apm-serverless-lambda-transactions/336984/2 "2023-06-27T16:25:26Z")

</div>

Hello again, @Kesha_Shah.

> [@Kesha\_Shah](#):
>
> I still tried adding environment variable `ELASTIC_APM_CAPTURE_BODY: transactions` but with that my transactions stopped coming to APM with below debug message

Really? I hope that adding `ELASTIC_APM_CAPTURE_BODY: transactions` does not _cause a change in whether there is an active transaction_! That `no active transaction found - cannot build new span` message happens when there is an attempt to create a tracing "span" (either in auto-instrumentations or by explicitly calling the [`startSpan` API](https://www.elastic.co/guide/en/apm/agent/nodejs/current/agent-api.html#apm-start-span)) but there is no "current transaction".

> [@Kesha\_Shah](#):
>
> but I also want to add event body for my transactions at least for errors.

To make sure I understand, you want some (or all) of the Lambda SQS event data to be added to the APM transaction so it is stored in Elasticsearch? If so, you should be able to use [`apm.setCustomContext()`](https://www.elastic.co/guide/en/apm/agent/nodejs/current/agent-api.html#apm-set-custom-context) to add unindexed data to a transaction, or one of the [API methods to add **labels**](https://www.elastic.co/guide/en/apm/agent/nodejs/current/agent-api.html#apm-set-label) which _are_ indexed. If you do use labels, please note the warnings about possible mapping explosions if you have too many labels -- see [Metadata | APM User Guide [8.11] | Elastic](https://www.elastic.co/guide/en/apm/guide/current/data-model-metadata.html).

---

<div class="post-metadata">

**Author:** ![Kesha\_Shah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kesha_shah/32/122625_2.png) [@Kesha\_Shah](https://discuss.elastic.co/u/Kesha_Shah)\
**Post date:** [June 28, 2023, 12:51pm UTC](https://discuss.elastic.co/t/not-able-to-find-a-way-to-add-event-json-in-apm-serverless-lambda-transactions/336984/3 "2023-06-28T12:51:56Z")

</div>

Hi @trentm

This might be a little long message!

I am getting very flaky behavior with APM for my serverless node js application which created more confusion in understanding of its working. Will try to mention one by one point here with my observations.

1. `no active transaction found - cannot build new span` - For the same code deployment hardly couple of times APM was able to create transaction and rest it gave us this debug message. When I compared both logs (Created transactions vs no active transaction) I found out that there is only one change in log which is `"message": "shimming handler@14.21.3 module"`, If I get this module in log then it will create transaction otherwise it will not. I have tried to trigger lambda from SQS message also I tried sending "Test Event" from AWS lambda UI but for both only couple of times it worked and rest it didn't work (No Transactions created and was not able to see lambda listed in APM)

2. APM Agent configuration coming in logs - I have followed exact [documentation](https://www.elastic.co/guide/en/apm/agent/nodejs/current/lambda.html) and also referred [sample](https://github.com/elastic/aws-lambda-apm-sample-app) but in my case for below configuration its not picking up serverUrl properly.

```auto
provider:
  stackTags:
    Department: product
    Purpose: production
    Environment: ${opt:stage,'local'}
  name: aws
  stage: ${opt:stage,'local'}
  runtime: nodejs14.x
  region: ap-south-1
  lambdaHashingVersion: 20201221
  memorySize: 512
  iamRoleStatements:
    - Effect: Allow
      Action:
        - lambda:InvokeFunction
      Resource: arn:aws:lambda:*:*:function:${self:custom.prefix}-worker
    - Effect: Allow
      Action:
        - sqs:DeleteMessage
        - sqs:ReceiveMessage
      Resource: arn:aws:sqs:*:*:${self:custom.prefix}-queue
  layers:
    - arn:aws:lambda:ap-south-1:<account_id>:layer:elastic-apm-extension-ver-1-4-0-x86_64:1
    - arn:aws:lambda:ap-south-1:<account_id>:layer:elastic-apm-node-ver-3-47-0:1
  environment:
    NODE_OPTIONS: -r elastic-apm-node/start
    ELASTIC_APM_LAMBDA_APM_SERVER: <cloud_server_url>
    ELASTIC_APM_SECRET_TOKEN: <secret_token>
    ELASTIC_APM_SEND_STRATEGY: background
    ELASTIC_APM_LOG_LEVEL: debug
    ELASTIC_APM_ENVIRONMENT: qa

```

This is printed configuration in logs :

```auto
{
    "log.level": "info",
    "@timestamp": "2023-06-28T10:45:55.693Z",
    "log": {
        "logger": "elastic-apm-node"
    },
    "agentVersion": "3.47.0",
    "env": {
        "pid": 17,
        "proctitle": "/var/lang/bin/node",
        "os": "linux 4.14.255-311-248.529.amzn2.x86_64",
        "arch": "x64",
        "host": "169.254.217.193",
        "timezone": "UTC+00",
        "runtime": "Node.js v14.21.3"
    },
    "config": {
        "serviceName": {
            "source": "default",
            "value": "qa-daily-report-worker",
            "commonName": "service_name"
        },
        "serviceVersion": {
            "source": "default",
            "value": "47",
            "commonName": "service_version"
        },
        "serverUrl": {
            "source": "default",
            "value": "http://127.0.0.1:8200/",
            "commonName": "server_url"
        },
        "logLevel": {
            "source": "environment",
            "value": "debug",
            "commonName": "log_level"
        },
        "environment": {
            "source": "environment",
            "value": "qa"
        },
        "secretToken": {
            "source": "environment",
            "value": "[REDACTED]",
            "commonName": "secret_token"
        }
    },
    "activationMethod": "aws-lambda-layer",
    "ecs": {
        "version": "1.6.0"
    },
    "message": "Elastic APM Node.js Agent v3.47.0"
}

```

if you see serverUrl here is `(http://127.0.0.1:8200/)` even though I added elastic cloud APM Url. If I add additional environment variable `ELASTIC_APM_SERVER_URL ` with my APM url then it shows up correctly in log. So do I need to add both environment variables (`ELASTIC_APM_SERVER_URL ,ELASTIC_APM_LAMBDA_APM_SERVER`) to setup correct APM?  
If I remove `ELASTIC_APM_LAMBDA_APM_SERVER` and just keep `ELASTIC_APM_SERVER_URL` then it gives error that Server Url can not be empty.

1. As per your last suggestion I want to add [customContext](https://www.elastic.co/guide/en/apm/agent/nodejs/current/agent-api.html#apm-set-custom-context) to capture events for each lambda invocations, but since we have added elastic-node-apm dependency as layer in serverless.yml I cant use that dependency directly in my handler code, so to do that I need to add `"elastic-apm-node": "^3.47.0"` in my `package.json` is it correct way to configure? Also since my serverless take care of starting APM agent I think I just need to retrieve APM agent and use it to set customContext as below.

```auto
export const dailyReport = async (event, context, callback) => {
  try {
     const apm = require('elastic-apm-node');     
     apm.setCustomContext(event);
    // More code ...
    return {
      statusCode: 200,
      body: 'ok'
    };
  } catch (err) {
    logger.error({message: err.message, stack: err.stack});
    callback(err);
  }

```

Since my APM is not creating transactions most of the times I am not able test my above code. So just confirming if its the right way to set context.

Let me know if I need to provide more logs 🙂

Thank you for your patience for reading such a long message!

---

<div class="post-metadata">

**Author:** ![trentm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trentm/32/77647_2.png) [@trentm](https://discuss.elastic.co/u/trentm)\
**Post date:** [June 28, 2023, 9:21pm UTC](https://discuss.elastic.co/t/not-able-to-find-a-way-to-add-event-json-in-apm-serverless-lambda-transactions/336984/4 "2023-06-28T21:21:14Z")

</div>

> [@Kesha\_Shah](#):
>
> When I compared both logs (Created transactions vs no active transaction) I found out that there is only one change in log which is `"message": "shimming handler@14.21.3 module"`, If I get this module in log then it will create transaction otherwise it will not.

The `shimming handler@14.21.3 module` is saying that the APM agent successfully found and is instrumenting your Lambda handler. The default Lambda function handler value is `index.handler`, which is where the "handler" in that message is coming from. I am guessing that the "version" in your package.json is "14.21.3".

We need to figure out why, in some cases, you do not get that message. There must be something different in those cases. Can you think of any possible differences in those deployments?

- If you could dump the contents of `process.env` in your handler function, that would help. I'm interested in `AWS_LAMBDA_FUNCTION_NAME`, `_HANDLER`, and `LAMBDA_TASK_ROOT`.
- If you look at your Lambda function in the AWS Console, what files are in your function and what is your "Handler" value (in the "Code properties" section):

 ![Screenshot 2023-06-28 at 2.11.05 PM](https://us1.discourse-cdn.com/elastic/original/3X/2/d/2d2f48da16a0167dac63feef6ae3134ac75738ae.png)

> [@Kesha\_Shah](#):
>
> if you see serverUrl here is `(http://127.0.0.1:8200/)` even though I added elastic cloud APM Url. If I add additional environment variable `ELASTIC_APM_SERVER_URL ` with my APM url then it shows up correctly in log. So do I need to add both environment variables (`ELASTIC_APM_SERVER_URL ,ELASTIC_APM_LAMBDA_APM_SERVER`) to setup correct APM?  
> If I remove `ELASTIC_APM_LAMBDA_APM_SERVER` and just keep `ELASTIC_APM_SERVER_URL` then it gives error that Server Url can not be empty.

Seeing `serverUrl: http://127.0.0.1:8200` there is correct. What is actually happening is that:

- the APM agent running in your application is talking to `http://127.0.0.1:8200`, and
- the `elastic-apm-extension-ver-1-4-0-x86_64` layer is running an extension that is listening at that address. It is the extension that eventually sends out the tracing data. The extension uses the `ELASTIC_APM_LAMBDA_APM_SERVER` value.

> [@Kesha\_Shah](#):
>
> Also since my serverless take care of starting APM agent I think I just need to retrieve APM agent and use it to set customContext as below.

That's right. The `aws:...:layer:elastic-apm-node-ver-3-47-0:1` Lambda layer is providing the `elastic-apm-node` Node.js package and putting it on your node import path. So your doing `const apm = require('elastic-apm-node');` is correct.

> [@Kesha\_Shah](#):
>
> Let me know if I need to provide more logs

Setting `ELASTIC_APM_LOG_LEVEL=trace` might help. It'll produce a lot of output.

---

<div class="post-metadata">

**Author:** ![Kesha\_Shah](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kesha_shah/32/122625_2.png) [@Kesha\_Shah](https://discuss.elastic.co/u/Kesha_Shah)\
**Post date:** [July 5, 2023, 9:19am UTC](https://discuss.elastic.co/t/not-able-to-find-a-way-to-add-event-json-in-apm-serverless-lambda-transactions/336984/5 "2023-07-05T09:19:06Z")

</div>

Hi @trentm This issue is fixed. From log analysis we figured out that `_handler` was set to Lumigo Tracer as we are currently using Lumigo for lambda tracing. We disabled Lumigo and apm traces started coming in.

But we found few more issues with our express app regarding transaction and logging, I have created discussion [here](https://discuss.elastic.co/t/elastic-apm-node-issues-regarding-transaction-and-logging/337648).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 2, 2023, 9:19am UTC](https://discuss.elastic.co/t/not-able-to-find-a-way-to-add-event-json-in-apm-serverless-lambda-transactions/336984/6 "2023-08-02T09:19:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.