Kibana - APM UI - selected trace cannot be found

Elastic Observability APM
#1

Hi,

Elasticsearch - v7.10.1
Kibana - V7.10.1
APM Server - V7.10.1

For some of the transactions in APM UI - Kibana is not showing the trace information, but instead it is showing the following message

The selected trace cannot be found

image
image1171×457 15.3 KB

This is happening for some traces (trace.id).

Eg:
trace.id : "41aa516af6a4a36286fa52c3a4889b75"
Number of transactions - 179,161
Number of spans - 5,022

This trace involves multiple services

The following table shows the top transactions with the number of spans. One transaction has 500 spans and all other has 15 or less spans

image
image872×350 17.8 KB

Any help will be appreciated

#2

Hi @chandra2037! This could happen if:

  • you have a transaction/trace id in the query parameters in the URL that is not available in the current time range
  • there are no sampled transactions matching the selected bucket or the query parameters

Also, do you mean that one trace has 179k transactions? That seems suspicious. What kind of trace is it?

#3

@dgieselaar Thank you for your response.

When I filter on trace.id : "41aa516af6a4a36286fa52c3a4889b75" within last 3 days (this trace happened around 03/22/2021 4:33 AM EST), getting the results (listing all services) as shown below

image
image1885×802 71.6 KB

When I enter into any particular service (eg: "role"), by keeping the traceid filter and same time range, it is showing following transactions

image
image1606×430 29.7 KB

Now when I enter into any particular transaction, lets say SecurityApiController#getSecurity, again keeping the same filter and time range, traces are not showing up

image
image1749×847 39.7 KB

This trace belongs to a async process.

#4

Hi - My team is also facing the same problem, any suggestion on how to fix this issue "The selected trace cannot be found"? Pls advise.

Thanks,
Parani.

#5

Hi @chandra2037 and @dgieselaar - Tagging to see if you have any resolution for this issue as my team is also facing the same problem.

Many thanks in advance.

#6

@Parani_Kumar

Not really. For us, this issue is happening for some of the traces.
what about you? Is this happening for all the traces/transactions?

#7

Hi @chandra2037 - Some traces for us as well, there are services which provides full transaction and traces. I had requested my team to generate more traffic for the same service which we are encountering "selected trace cannot be found" msg. Not sure how far this will help but this is happening only for some transactions. I will keep you posted the outcome. Thanks for your response!

Thanks,
Parani.