Separate transaction for every resolver in AWS Lambda

Hello! I'm using Elastic APM v8.15.0 with an AWS Lambda function written in Node.js, integrated via elastic-apm-node v4.7.3. My Lambda function has three separate resolvers. Here’s my current configuration setup:


locals {
  lambda_fn_name                            = "${var.environment}_my_lambda_resolver"
  elastic_apm_send_strategy                 = "background"
  elastic_apm_verify_server_cert            = "false"
  elastic_apm_service_name                  = "my_lambda_service"
  elastic_apm_enabled                       = "true"
}

resource "aws_lambda_function" "my_lambda_function" {
  function_name = local.lambda_fn_name
  publish       = true
  role          = xyz
  image_uri     = xyz
  package_type  = "xyz"
  timeout       = 120

  environment {
    variables = {
      ELASTIC_APM_ENABLED                   = local.elastic_apm_enabled
      ELASTIC_APM_SERVICE_NAME              = local.elastic_apm_service_name
      ELASTIC_APM_SERVER_URL                = xyz
      ELASTIC_APM_API_KEY                   = xyz
      ELASTIC_APM_GLOBAL_LABELS             = true
      ELASTIC_APM_SEND_STRATEGY             = local.elastic_apm_send_strategy
      ELASTIC_APM_LAMBDA_VERIFY_SERVER_CERT = local.elastic_apm_verify_server_cert
      ELASTIC_APM_ENVIRONMENT               = var.environment
      NODE_OPTIONS                          = "-r elastic-apm-node/start"
    }
  }
}

I can see my Lambda function listed in the APM > Services section, but I’m running into three issues:

  1. Separate Transactions per Resolver: Currently, only one transaction is shown per environment. Each resolver function’s calls are grouped under a single transaction, so I can’t differentiate which resolver each trace represents. I attempted to create individual transactions and spans within each resolver, but that resulted in losing the detailed trace samples, leaving only the initial call.

Here’s an example of the resolver setup:


const myResolverFunction: ResolverHandler<StatusArgs, StatusResult> = async (event, _context, _callback) => {
  const transaction = apm.startTransaction('myResolverFunction', 'resolver');

  try {
    const span = apm.startSpan('my_span');
    const data = await apiClient.runMyResolverOperations();
    span?.end();

    transaction.result = 'success';
    return {
      __typename: 'TaskBatch',
      ...data,
    };
  } catch (error: unknown) {
    transaction.result = 'error';
    if (error instanceof Error) {
      apm.captureError(error);
    } else {
      apm.captureError(String(error));
    }
    throw error;
  } finally {
    transaction.end();
  }
};
  1. Trace Sample Order: The trace samples appear in a seemingly random order, which complicates analysis. I’d like them sorted by the time they were called.

  2. Viewing API Request Bodies: While I can see all API calls made, the request bodies aren’t visible. Is there a way to configure APM to capture and display the full request body of each call?

Any guidance on achieving these three goals in my Elastic APM setup would be greatly appreciated! Thank you!

Hi @mkaniaa. By "resolvers" do you mean using AWS AppSync (AWS's thing for building GraphQL services)? AWS AppSync JavaScript resolver function reference for Lambda - AWS AppSync

I haven't used this myself.

I'm struggling a little bit to understand the architecture of your app/service. Are you able to show more code for context, perhaps giving a name for each of the three resolvers so our conversation can be a little bit more concrete.

It is possible you would be able to use the transaction.name (Transaction API | APM Node.js Agent Reference [4.x] | Elastic) API to tweak the current transaction name to something more meaningful for each different type of call to your Lambda function. I don't know from your description if the resolver name or some GraphQL method name would make sense.

Do you mean in the service "Transactions" view, e.g.:

I'm not sure if there is a way to control sorting here. In the general case this is a sampling of all transactions for the service for the current time range, limited to 500. I had thought they would show up sorted by transaction start time, but I don't know that that's the case.

The Configuration options | APM Node.js Agent Reference [4.x] | Elastic (captureBody) config var exists to capture the HTTP body of incoming requests, but there is no support for capturing the body of outgoing requests, unfortunately.

Hello @trentm, I really appreciate your reply! According to your questions:

  1. Yes, by “resolvers” I mean using AWS AppSync. I have one main AppSyncResolverHandler and a map of 3 resolvers (I simplified the names for this example). The app.ts file is quite simple - it just awaits all the resolvers:
const resolverMap: ResolverMap = {
  'Query::getTableColumnData': getTableColumnValues,
  'Mutation::updateStatus': updateStatus,
  'Query::getSendToData': getSendToValues,
};

export const handler: AppSyncResolverHandler<any, any> = async (
  event: AppSyncResolverEvent<any, any>,
  context: Context,
  callback: Callback
) => {
  const { fieldName, parentTypeName } = event.info;

  logEvent(event);
  loadAwsConfig();

  const resolver = getResolver({ fieldName, parentTypeName, resolverMap });
  if (resolver) {
    try {
      return await resolver(event, context, callback);
    } catch (e) {
      console.error('Error:', e);
      return errorMapper(e as string | AxiosError<{ detail: string }>);
    }
  } else {
    const errorMsg = `No handler for field '${fieldName}' of type '${parentTypeName}'`;
    return errorMapper(errorMsg);
  }
};
  1. Yes, this is exactly the "Transactions" view I meant. You can be right - they're are sorted by the call time. I'm wondering if it's possible to show them on some timeline? Currently, I can see only the latency graph, so again - the investigation is hard to perform, when you look for some specific calls that you made for testing.
  2. Thank you for the idea of using captureBody - I will try it for sure.

@trentm, I tried out the solutions you suggested:

  1. Setting transaction.name at the start of each resolver did the trick! Now, I can see separate transaction summaries for each resolver - thanks for the tip!

  2. Following the documentation, I set ELASTIC_APM_CAPTURE_BODY to "all". I do see a new attribute, http.request.body.original, for some requests, but the value is consistently [REDACTED]. Do you know if this might be due to something in my infrastructure or if there’s another Elastic configuration I might need to adjust? It seems like setting ELASTIC_APM_CAPTURE_BODY to "all" might not be enough.

  3. I’m still looking for a way to view all transactions for a given resolver on a timeline. Is there a way to add a custom widget to the transaction view to display them on a timeline? Right now, I only see the widget related to latency.

Thanks for your help!

(Again now that captureBody is only about capturing the HTTP body for incoming requests to your service.)

Hrm, getting [REDACTED] should not happen if it is set to 'all'. Perhaps something is surprisingly setting that value to something other than 'all'.

  • First thing to check: At startup, the APM agent emits a log line (at "info" level) that includes a dump of its config. E.g.:
{"log.level":"info","@timestamp":"2024-10-29T20:21:56.138Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","agentVersion":"4.8.0","env":{"pid":12004,"proctitle":"node","os":"darwin 23.6.0","arch":"arm64","host":"peach.local","timezone":"UTC-0700","runtime":"Node.js v18.20.4"},"config":{"logLevel":{"source":"default","value":"info","commonName":"log_level"},"serverUrl":{"source":"default","value":"http://127.0.0.1:8200/","commonName":"server_url"},"serviceName":{"source":"start","value":"example-trace-http-request","commonName":"service_name"},"serviceVersion":{"source":"default","value":"4.8.0","commonName":"service_version"}},"activationMethod":"require","message":"Elastic APM Node.js Agent v4.8.0"}

Can you find that to see if you have a value for captureBody in that log line?

Central config success: updated captureBody: ${value}

The built-in APM UI doesn't support this. You'll have to poke around in Discover (in Kibana) or build your own Dashboard.

For example in Discover, searching the traces-apm* data stream with the query service.name: "example-trace-http-request" AND processor.event:"transaction" (for my example service) looks something like this:

That shows individual transactions, and a chart at the top that shows the number of transactions per time bucket. There are some limited options on that chart provide a breakdown (perhaps by transaction.name).

If that is not sufficient, then you could look at the Dashboards in Kibana and play around with a visualization that works for you:

The charting support is very powerful, which is good, but means there is a learning curve.

Hello @trentm, here’s the log you requested (with certain addresses altered due to policy):

{
    "log.level": "info",
    "@timestamp": "2024-10-29T18:23:10.171Z",
    "log.logger": "elastic-apm-node",
    "ecs.version": "8.10.0",
    "agentVersion": "4.7.3",
    "env": {
        "pid": 8,
        "proctitle": "/var/lang/bin/node",
        "os": "linux 5.10.226-235.879.amzn2.x86_64",
        "arch": "x64",
        "host": "111.111.11.111",
        "timezone": "UTC+00",
        "runtime": "Node.js v20.18.0"
    },
    "config": {
        "captureBody": {
            "source": "environment",
            "value": "all",
            "commonName": "capture_body"
        },
        "environment": {
            "source": "environment",
            "value": "dev"
        },
        "globalLabels": {
            "source": "environment",
            "value": [
                [
                    "service_namespace",
                    "service_dev"
                ]
            ],
            "sourceValue": "service_namespace=service_dev"
        },
        "logLevel": {
            "source": "default",
            "value": "info",
            "commonName": "log_level"
        },
        "serverUrl": {
            "source": "environment",
            "value": "https://example-server.com/",
            "commonName": "server_url"
        },
        "apiKey": {
            "source": "environment",
            "value": "[REDACTED]",
            "commonName": "api_key"
        },
        "serviceName": {
            "source": "environment",
            "value": "fsn_appsync_batch_smart_queue",
            "commonName": "service_name"
        },
        "serviceVersion": {
            "source": "default",
            "value": "$LATEST",
            "commonName": "service_version"
        }
    },
    "activationMethod": "env-attach",
    "message": "Elastic APM Node.js Agent v4.7.3"
}

(Again now that captureBody is only about capturing the HTTP body for incoming requests to your service.)

I'm trying to understand, how it apply to my case, so maybe I will clarify my setup and goals, here’s what I’d like to achieve and please let me know if it's possible with Elastic APM.

  1. Tracking event.arguments.input for Incoming Lambda Events
    My AppSync Lambda function is triggered by events containing parameters that my resolvers need for various actions. These parameters are nested in event.arguments.input, as shown here:
export const getValues: AppSyncResolverHandler<
  DataArgs,
  ValuesResult
> = async (event, _context, _callback) => {
  apm.setTransactionName('myTransactionName');
  const { columns, teamId, userId } = event.arguments.input;

I’d like to see the content of event.arguments.input in APM to review the actual values that triggered the resolver. Currently, I can only see event.outcome and event.success_count.

  1. Capturing Request and Response Bodies for Outgoing API Calls
    My resolver functions use the incoming event values to make further API requests. Ideally, I’d like to capture both the request body and response content of these outgoing calls. Another example from the code:
documents = await documentApiClient.getDocuments(documentIds);

Right now, I see only:

  • http.request.body.original always with the value [REDACTED].
  • http.request.env showing values for REMOTE_ADDR, SERVER_NAME, and SERVER_PORT.
  • http.request.headers with the correct header values.
  • http.request.method for the request method used.
  • http.response.headers with response headers from the API.
  • http.response.status_code with the API response status code.

As you can see, the request and response bodies, which are crucial for my investigation, are not accessible or are redacted.

Any insights on whether it’s possible to achieve these goals with Elastic APM would be very helpful. Thank you!

Ad 1:
In the traces for specific APIs, I can see the URL arguments used for the requests, which is sufficient for my needs.

Ad 2:

  • I had to set the flag ELASTIC_APM_CAPTURE_BODY to "all" for each API called by AppSync to view the request bodies for each call.
  • I was able to capture response bodies by using labels.

My remaining concern is the maxLength for JSON in labels. Here’s how I’m currently creating the label for responses in my Flask API service:

@app.after_request
def apm_label_response_data(response):
    response_data = response.get_json(silent=True)
    limited_data = json.dumps(response_data)[:10000]
    elasticapm.label(response_body=limited_data)
        
    return response

The problem is that my limit doesn’t seem to take effect, as the JSON is still truncated to only 1024 characters. I believe the reason is explained here. Is there any way to adjust this limit for my label?

I created a separate topic for the maxLength JSON issue: Max Characters Length for JSON in transaction labels.

The problem described here is solved and this topic can be closed.