Transaction result setting as failure/spindown

gabriel-f-santos
Wed, 13 Sep 2023 12:28:12 +0000
APM

Hello everyone!

We are using agent and APM extensions for monitoring our serverless application with Python Lambdas. We're following this configuration for SAM applications:

APM Agent language and version :

Language: python running in aws lambda.

arn:aws:lambda:us-east-1:267093732750:layer:elastic-apm-extension-ver-1-4-0-x86_64:1

arn:aws:lambda:us-east-1:267093732750:layer:elastic-apm-python-ver-6-18-0:1

Our environment is configured as follows:

se********rvice.environment | ***
service.framework.name | AWS Lambda
service.language.name | python
service.language.version | 3.9.17
service.runtime.name | AWS_Lambda_python3.9
service.runtime.version | 3.9.17
service.version | $LATEST

Environment Variables:

ELASTIC_APM_SERVICE_NAME: SERVICE_NAME
ELASTIC_APM_SEND_STRATEGY: background
AWS_LAMBDA_EXEC_WRAPPER: /opt/python/bin/elasticapm-lambda
ELASTIC_APM_SECRET_TOKEN: ELASTIC_TOKEN
ELASTIC_APM_LAMBDA_APM_SERVER: ELASTIC_URL
ELASTIC_APM_ENVIRONMENT: ENVIRONMENT

Fresh install or upgraded from other version?

We observed this error after change from configuration elastic-apm python lib + lambda extension to lambda extension for both (elastic apm python and extension).

Is there anything special in your setup?

Description of the problem including expected versus actual behavior.

However, we are facing a situation:

The Lambda works normally and does what it's supposed to do. However, the transaction is being marked as failed.

The configuration we'll describe in the message below consists of an SQS queue consumed by a Lambda function. We tried to reproduce this error by. As AWS spins up Lambdas to consume the data, we noticed this occurs when the transaction status is set to "spindown". According to the AWS extension documentation, the "spindown" event is emitted to the extensions when the Lambda is about to terminate its runtime (Usar a API de extensões do Lambda para criar extensões - AWS Lambda).

I would like some assistance in understanding if setting this value as the transaction result is the expected outcome, or if there is an issue.

How we can see in bellow image, lambda works fine but some transactions is marked as failed and transaction.result as spindown.

Steps to reproduce :

1. Just add some messages to the queue (we used 200 messages)

Metada ta :

transaction

transaction.duration.us | 39998000
transaction.id | ************
transaction.message.age.ms | 91
transaction.message.queue.name | ************
transaction.name | ************
transaction.name.text | ************
transaction.representative_count | 1
transaction.result | spindown
transaction.sampled | true
transaction.span_count.dropped | 0
transaction.span_count.started | 0
transaction.type | messaging

event

event.agent_id_status | missing
event.ingested | 2023-09-12T17:45:32Z
event.outcome | failure
event.success_count | 0

Usefull logs from cloudwatch :

Environment variable:

`` ELASTIC_APM_LOG_LEVEL: trace

gist.github.com

https://gist.github.com/gabriel-f-santos/80b0aa87cde267626b5af9aa34d33f1a

log.txt

| INIT_START Runtime Version: python:3.9.v30	Runtime Version ARN: arn:aws:lambda:::runtime:86d4ce088432216337acec891c716c30002d0ed911f5a9574e36052e7527d6ab |
| {"log.level":"debug","@timestamp":"2023-09-12T17:29:55.189Z","log.origin":{"file.name":"extension/client.go","file.line":123},"message":"ExtensionID : ab65a2f9-3804-41ff-b5f0-e46f176d893f","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:29:55.189Z","log.origin":{"file.name":"app/run.go","file.line":45},"message":"Register response: {\n\t\"functionName\": \"send-default\",\n\t\"functionVersion\": \"$LATEST\",\n\t\"handler\": \"handlers.send_notification\"\n}","ecs.version":"1.6.0"} |
| {"log.level":"info","@timestamp":"2023-09-12T17:29:55.190Z","log.origin":{"file.name":"apmproxy/receiver.go","file.line":57},"message":"Extension listening for apm data on :8200","ecs.version":"1.6.0"} |
| {"log.level":"info","@timestamp":"2023-09-12T17:29:55.194Z","log.origin":{"file.name":"logsapi/subscribe.go","file.line":73},"message":"Extension listening for Lambda Logs API events on 169.254.79.130:34301","ecs.version":"1.6.0"} |
| LOGS Name: apm-lambda-extension State: Subscribed Types: [Platform, Function] |
| {"log.level":"info","@timestamp":"2023-09-12T17:29:55.198Z","log.origin":{"file.name":"app/run.go","file.line":127},"message":"Waiting for next event...","ecs.version":"1.6.0"} |
| [DEBUG] 2023-09-12T17:29:55.561Z  Inserting elasticapm log_record_factory into logging |
| [DEBUG] 2023-09-12T17:29:55.598Z  Detected PID change from None to 17, starting threads |
| [DEBUG] 2023-09-12T17:29:55.598Z  Starting transport thread |
This file has been truncated. show original

Thanks for any help.

2 posts - 2 participants