Hello everyone!
We are using agent and APM extensions for monitoring our serverless application with Python Lambdas. We're following this configuration for SAM applications:
APM Agent language and version :
Language: python running in aws lambda.
- arn:aws:lambda:us-east-1:267093732750:layer:elastic-apm-extension-ver-1-4-0-x86_64:1
- arn:aws:lambda:us-east-1:267093732750:layer:elastic-apm-python-ver-6-18-0:1
Our environment is configured as follows:
| se********rvice.environment | *** |
|---|---|
| service.framework.name | AWS Lambda |
| service.language.name | python |
| service.language.version | 3.9.17 |
| service.runtime.name | AWS_Lambda_python3.9 |
| service.runtime.version | 3.9.17 |
| service.version | $LATEST |
Environment Variables:
ELASTIC_APM_SERVICE_NAME: SERVICE_NAME
ELASTIC_APM_SEND_STRATEGY: background
AWS_LAMBDA_EXEC_WRAPPER: /opt/python/bin/elasticapm-lambda
ELASTIC_APM_SECRET_TOKEN: ELASTIC_TOKEN
ELASTIC_APM_LAMBDA_APM_SERVER: ELASTIC_URL
ELASTIC_APM_ENVIRONMENT: ENVIRONMENT
Fresh install or upgraded from other version?
We observed this error after change from configuration elastic-apm python lib + lambda extension to lambda extension for both (elastic apm python and extension).
Is there anything special in your setup?
Description of the problem including expected versus actual behavior.
However, we are facing a situation:
The Lambda works normally and does what it's supposed to do. However, the transaction is being marked as failed.
The configuration we'll describe in the message below consists of an SQS queue consumed by a Lambda function. We tried to reproduce this error by. As AWS spins up Lambdas to consume the data, we noticed this occurs when the transaction status is set to "spindown". According to the AWS extension documentation, the "spindown" event is emitted to the extensions when the Lambda is about to terminate its runtime (API Extensions do Lambda - AWS Lambda).
I would like some assistance in understanding if setting this value as the transaction result is the expected outcome, or if there is an issue.
How we can see in bellow image, lambda works fine but some transactions is marked as failed and transaction.result as spindown.
Steps to reproduce :
- Just add some messages to the queue (we used 200 messages)
Metada ta :
transaction
| transaction.duration.us | 39998000 |
|---|---|
| transaction.id | ************ |
| transaction.message.age.ms | 91 |
| transaction.message.queue.name | ************ |
| transaction.name | ************ |
| transaction.name.text | ************ |
| transaction.representative_count | 1 |
| transaction.result | spindown |
| transaction.sampled | true |
| transaction.span_count.dropped | 0 |
| transaction.span_count.started | 0 |
| transaction.type | messaging |
event
| event.agent_id_status | missing |
|---|---|
| event.ingested | 2023-09-12T17:45:32Z |
| event.outcome | failure |
| event.success_count | 0 |
Usefull logs from cloudwatch :
Environment variable:
`` ELASTIC_APM_LOG_LEVEL: trace
Thanks for any help.