APM does not display spans despite having them in ES

Dzikidzik
Mon, 12 Feb 2024 10:04:45 +0000
APM

I have two Django apps and celery running tasks in background, and Kibana seems to not display properly spans for transaction that are stored in ES.

Transaction metadata shows transaction.span_count.started: 48 and indeed when I search for this transaction.id in Discover I see 49 documents

In this particular case, I had a request to service A, that send a request to service B, which scheduled about 1500 celery tasks (celery task transaction shown above). When I click on View full trace it shows the transaction of service A, but spans are not displayed

Transaction metadata shows transaction.span_count.started: 5 which means it only counts spans from service A, that makes sense. It does not however display transaction and spans from service B, which is stored in ES. I need to find the transaction.id from Discover view and filter service B transactions in APM only to see the same thing

This time transaction.span_count.started: 100 and transaction.span_count.dropped: 1557 as I have a 100 spans per transaction limit, everything as expected. Everything except for displaying those spans correctly in APM UI

Discover says I have 109k documents with the same trace.id, thats a bit more than I would expect but still, the data is there, it's just not displayed properly. It does seem to properly aggregate the span data tho, as Time spent by span type propery displays different span types that were created in this trace (I'd love to add a screen here but I can only add one media in the post)

This however does not happen regularly. In like 99% of cases when I browse APM logs for regular HTTP requests everything works fine. In like 10% of cases when I browse APM logs for celery tasks, this happens. I can easily find other examples when similar trace of Service A -> Service B -> Celery task renders correctly

Kibana version:
7.17.7

Elasticsearch version:
7.17.7

APM Server version:
7.17.7

APM Agent language and version:
Python 6.13.2

Browser version:
Chrome 119.0.6045.159

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
n/a

Fresh install or upgraded from other version?
Fresh

Is there anything special in your setup?
n/a

Steps to reproduce:
I couldnt reliably reproduce the issue

Errors in browser console (if relevant):

GET https://kibana.[...].com/internal/apm/settings/anomaly-detection/jobs 403 (Forbidden)
Unchecked runtime.lastError: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received

Not sure if those are related

Provide logs and/or server output (if relevant):

15 posts - 3 participants