Kibana version:
elastic cloud 7.3.2
APM Agent language and version:
dotnet 1.5.1
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
nuget package
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I deployed a new dotnet core 3.1 function in AWS Lambda and i noticed barely any, if any at all, traces land in APM
I tried setting crazy environment variables like ELASTIC_APM_FLUSH_INTERVAL=50ms, ELASTIC_APM_MAX_BATCH_EVENT_COUNT=1 and adding a 200ms delay before exiting my lambda call to no avail.
Enabling debug log with ELASTIC_APM_LOG_LEVEL=Debug, the last message logged is always {PayloadSenderV2} Enqueued Transaction
I have python APM Agents deployed on AWS Lambda that seem to work super fine and a dotnet APM Agent deployed in ECS that also works fine, the problem seems to only be with dotnet Agent + AWS Lambda
I'm convinced a method for manually and synchronously flushing the APM Agent send queue before exiting lambda invocations is required here, but none is found
Steps to reproduce:
- Deploy a function in AWS Lambda
- Watch in despair as your traces never land in APM
Provide logs and/or server output (if relevant):
2020-06-06 18:16:40.2229|Info|61|hotspring.Connector.RowsCopied|trace.id=9a67f4b8e5f8fd44b50f12dc4be22a21, transaction.id=84b7ee70ad438d9a, AWSRequestId=9fe6c424-8e0e-5288-b24b-937151bc1dab, batchId=hotspring/manifest/4896348d-fcef-486c-96af-8bedf647ea88, fileCount=3, fileLines=5, durationInMS=537|Copy statistics
[2020-06-06 18:16:40.224 +00:00][Debug] - {PayloadSenderV2} Enqueued Span. newEventQueueCount: 130. MaxQueueEventCount: 1000. Span: Span{Id: 15f9fe945acca95e, TransactionId: 84b7ee70ad438d9a, ParentId: 84b7ee70ad438d9a, TraceId: 9a67f4b8e5f8fd44b50f12dc4be22a21, Name: Connector.RunCopy, Type: redshift, IsSampled: True}.
2020-06-06 18:16:40.2229|Info|61|hotspring.Batch.Execute|trace.id=9a67f4b8e5f8fd44b50f12dc4be22a21, transaction.id=84b7ee70ad438d9a, AWSRequestId=9fe6c424-8e0e-5288-b24b-937151bc1dab, batch_id=4896348d-fcef-486c-96af-8bedf647ea88|Executed
[2020-06-06 18:16:40.242 +00:00][Debug] - {PayloadSenderV2} Enqueued Span. newEventQueueCount: 131. MaxQueueEventCount: 1000. Span: Span{Id: 33e45ccfa12b02ae, TransactionId: 84b7ee70ad438d9a, ParentId: 84b7ee70ad438d9a, TraceId: 9a67f4b8e5f8fd44b50f12dc4be22a21, Name: dynamo.SetBatchStatus, Type: dynamodb, IsSampled: True}.
[2020-06-06 18:16:40.320 +00:00][Debug] - {PayloadSenderV2} Enqueued Span. newEventQueueCount: 132. MaxQueueEventCount: 1000. Span: Span{Id: 295de3bcfc6c6949, TransactionId: 84b7ee70ad438d9a, ParentId: 84b7ee70ad438d9a, TraceId: 9a67f4b8e5f8fd44b50f12dc4be22a21, Name: s3Client.DeleteObjectAsync, Type: s3, IsSampled: True}.
[2020-06-06 18:16:40.337 +00:00][Debug] - {PayloadSenderV2} Enqueued Span. newEventQueueCount: 133. MaxQueueEventCount: 1000. Span: Span{Id: d345c807c2639896, TransactionId: 84b7ee70ad438d9a, ParentId: 84b7ee70ad438d9a, TraceId: 9a67f4b8e5f8fd44b50f12dc4be22a21, Name: dynamo.SetBatchStatus, Type: dynamodb, IsSampled: True}.
[2020-06-06 18:16:40.338 +00:00][Debug] - {PayloadSenderV2} Enqueued Span. newEventQueueCount: 134. MaxQueueEventCount: 1000. Span: Span{Id: fe8887841dc31dbf, TransactionId: 84b7ee70ad438d9a, ParentId: 84b7ee70ad438d9a, TraceId: 9a67f4b8e5f8fd44b50f12dc4be22a21, Name: batch.ExecuteBatch, Type: exec, IsSampled: True}.
[2020-06-06 18:16:40.338 +00:00][Debug] - {PayloadSenderV2} Enqueued Transaction. newEventQueueCount: 135. MaxQueueEventCount: 1000. Transaction: Transaction{Id: 84b7ee70ad438d9a, TraceId: 9a67f4b8e5f8fd44b50f12dc4be22a21, ParentId: null, Name: SQS-Execute, Type: exec, IsSampled: True}.
END RequestId: 9fe6c424-8e0e-5288-b24b-937151bc1dab
REPORT RequestId: 9fe6c424-8e0e-5288-b24b-937151bc1dab Duration: 2356.85 ms Billed Duration: 2400 ms Memory Size: 256 MB Max Memory Used: 158 MB