Transaction result setting as failure/spindown

gabriel-f-santos · September 13, 2023, 12:28pm

Hello everyone!

We are using agent and APM extensions for monitoring our serverless application with Python Lambdas. We're following this configuration for SAM applications:

APM Agent language and version :

Language: python running in aws lambda.

arn:aws:lambda:us-east-1:267093732750:layer:elastic-apm-extension-ver-1-4-0-x86_64:1
arn:aws:lambda:us-east-1:267093732750:layer:elastic-apm-python-ver-6-18-0:1

Our environment is configured as follows:

se****rvice.environment	***
service.framework.name	AWS Lambda
service.language.name	python
service.language.version	3.9.17
service.runtime.name	AWS_Lambda_python3.9
service.runtime.version	3.9.17
service.version	$LATEST

Environment Variables:

ELASTIC_APM_SERVICE_NAME: SERVICE_NAME
ELASTIC_APM_SEND_STRATEGY: background
AWS_LAMBDA_EXEC_WRAPPER: /opt/python/bin/elasticapm-lambda
ELASTIC_APM_SECRET_TOKEN: ELASTIC_TOKEN
ELASTIC_APM_LAMBDA_APM_SERVER: ELASTIC_URL
ELASTIC_APM_ENVIRONMENT: ENVIRONMENT

Fresh install or upgraded from other version?

We observed this error after change from configuration elastic-apm python lib + lambda extension to lambda extension for both (elastic apm python and extension).

Is there anything special in your setup?

Description of the problem including expected versus actual behavior.

However, we are facing a situation:

The Lambda works normally and does what it's supposed to do. However, the transaction is being marked as failed.

The configuration we'll describe in the message below consists of an SQS queue consumed by a Lambda function. We tried to reproduce this error by. As AWS spins up Lambdas to consume the data, we noticed this occurs when the transaction status is set to "spindown". According to the AWS extension documentation, the "spindown" event is emitted to the extensions when the Lambda is about to terminate its runtime (API Extensions do Lambda - AWS Lambda).

I would like some assistance in understanding if setting this value as the transaction result is the expected outcome, or if there is an issue.

How we can see in bellow image, lambda works fine but some transactions is marked as failed and transaction.result as spindown.

Steps to reproduce :

Just add some messages to the queue (we used 200 messages)

Metada ta :

transaction

transaction.duration.us	39998000
transaction.id	************
transaction.message.age.ms	91
transaction.message.queue.name	************
transaction.name	************
transaction.name.text	************
transaction.representative_count	1
transaction.result	spindown
transaction.sampled	true
transaction.span_count.dropped	0
transaction.span_count.started	0
transaction.type	messaging

event

event.agent_id_status	missing
event.ingested	2023-09-12T17:45:32Z
event.outcome	failure
event.success_count	0

Usefull logs from cloudwatch :

Environment variable:

`` ELASTIC_APM_LOG_LEVEL: trace

gist.github.com

https://gist.github.com/gabriel-f-santos/80b0aa87cde267626b5af9aa34d33f1a

log.txt

| INIT_START Runtime Version: python:3.9.v30	Runtime Version ARN: arn:aws:lambda:::runtime:86d4ce088432216337acec891c716c30002d0ed911f5a9574e36052e7527d6ab |
| {"log.level":"debug","@timestamp":"2023-09-12T17:29:55.189Z","log.origin":{"file.name":"extension/client.go","file.line":123},"message":"ExtensionID : ab65a2f9-3804-41ff-b5f0-e46f176d893f","ecs.version":"1.6.0"} |
| {"log.level":"debug","@timestamp":"2023-09-12T17:29:55.189Z","log.origin":{"file.name":"app/run.go","file.line":45},"message":"Register response: {\n\t\"functionName\": \"send-default\",\n\t\"functionVersion\": \"$LATEST\",\n\t\"handler\": \"handlers.send_notification\"\n}","ecs.version":"1.6.0"} |
| {"log.level":"info","@timestamp":"2023-09-12T17:29:55.190Z","log.origin":{"file.name":"apmproxy/receiver.go","file.line":57},"message":"Extension listening for apm data on :8200","ecs.version":"1.6.0"} |
| {"log.level":"info","@timestamp":"2023-09-12T17:29:55.194Z","log.origin":{"file.name":"logsapi/subscribe.go","file.line":73},"message":"Extension listening for Lambda Logs API events on 169.254.79.130:34301","ecs.version":"1.6.0"} |
| LOGS Name: apm-lambda-extension State: Subscribed Types: [Platform, Function] |
| {"log.level":"info","@timestamp":"2023-09-12T17:29:55.198Z","log.origin":{"file.name":"app/run.go","file.line":127},"message":"Waiting for next event...","ecs.version":"1.6.0"} |
| [DEBUG] 2023-09-12T17:29:55.561Z  Inserting elasticapm log_record_factory into logging |
| [DEBUG] 2023-09-12T17:29:55.598Z  Detected PID change from None to 17, starting threads |
| [DEBUG] 2023-09-12T17:29:55.598Z  Starting transport thread |

This file has been truncated. show original

Thanks for any help.

system · October 11, 2023, 12:28pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Failed to submit message (aws lambda using python) APM python	4	733	June 5, 2023
Elastic APM AWS Lambda Python APM docker , python	14	1041	August 8, 2022
Not able to find a way to add event.json in APM serverless Lambda transactions APM nodejs	5	487	August 2, 2023
How to start and end transaction in python aws lambda function , also how to add labels between transaction APM python , aws-lambda	17	1231	March 14, 2023
Transcations are not coming from lambda-django function APM python	4	509	November 5, 2020

Transaction result setting as failure/spindown

transaction

event

Related topics