APM UI - Not able to search for child span/transaction labels

[eduhenke]

Kibana version: 7.15.1

Elasticsearch version: 7.15.1

APM Server version: 7.15.1

APM Agent language and version: Node.JS at 3.26.0

Browser version: Chrome

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Docker

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup? Not that I can think of, just a simple docker-compose

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I expected to be able to filter traces based on their transactions' labels. However as shown below I am not able to do that.

With the following distributed transaction:

When I see the child transaction from the "station-write" service, we can see that there is a "stationId" label:

When I see the parent transaction from the "gateway" service, we can see that there are other labels:

Steps to reproduce:

1. Go to "Traces" page, enter in the search box the "labels.stationId" field(we can see that the auto complete works):
   

   

2. Filter by a given value, and see that it doesn't show any transactions
   

   

I couldn't find anywhere that said that it is(or is not) possible to search by labels in child spans/transactions. If it isn't possible to search by those labels, is there any way that I can add those labels to the parent transaction, in order for them to be searchable? The gateway service starts the trace, and this "station-write" service creates another transaction inside of that distributed trace(they are in different pods).

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

[eduhenke]

this is still a problem for me

[dgieselaar (Dario Gieselaar)]

@eduhenke it is indeed not possible to search by child spans. One workaround we have today is to make sure those labels are added to the transaction as well. Alternatively, you can use Discover to look for spans that match your search, and copy the trace/transaction id. You can then use those as a filter in the APM UI.

[eduhenke]

Hi @dgieselaar, thanks for answering!

I understood that I can't search by the child spans, however the search also doesn't work with distributed transactions. It is from my understanding that the child transaction created in a microservice and linked with another parent transaction(maybe coming from a gateway in front of that microservice), is indeed a transaction. However I can't search for that child transaction as well. Is that behavior also expected?

[dgieselaar (Dario Gieselaar)]

@eduhenke

I understood that I can't search by the child spans, however the search also doesn't work with distributed transactions. It is from my understanding that the child transaction created in a microservice and linked with another parent transaction(maybe coming from a gateway in front of that microservice), is indeed a transaction. However I can't search for that child transaction as well. Is that behavior also expected?

@eduhenke This is not possible in the traces overview, which only lists root transactions (ie transactions without a parent). However, in this scenario I think you could possibly use the services overview to filter for the child transaction. Would that work for you? (And apologies for the late reply, this dropped off my radar).