Span type for Full trace

[Harm]

Kibana version: 7.6.0

Elasticsearch version: 7.6.0

APM Server version: 7.6.0

APM Agent language and version: Java 1.12-1

Browser version: Chrome 80.0.3987.149

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Kibana from Elastic Cloud, Java agent from Maven, APM-server via .deb

Fresh install or upgraded from other version? Upgrade from 7.4.0

Is there anything special in your setup? -

Description of the problem including expected versus actual behavior: Our Java APM agent is setting span types like app or db. The transactions are correlated with frontend services, but these have no span types. When I select 'View Full Trace', the page shows the complete trace, but the JAVA backend span types are not visible anymore. "Time spent by span type" says: "No data within this time range."

Is this the expected behavior? I can imagine the span types of the backend services should still be visible when viewing the full trace?

Steps to reproduce:

1. Open a backend service
2. Select a transaction (Span types are visible)
3. Click on View Full Trace.
4. Span types are empty now

Errors in browser console (if relevant): None

Provide logs and/or server output (if relevant):

[Sylvain_Juge]

Hi @Harm,

When you say

Our Java APM agent is setting span types like app or db

Does it means that you are using API to set those explicitly or that agent default behavior for auto-instrumentation is used ?

Can you provide screenshots before clicking on 'show full trace' and after clicking on it ? That would help us understand what is missing, I haven't been able to reproduce on my side.

From what I understand this is a UI issue, not something related to the java agent, right ?

[Harm]

Hi @Sylvain_Juge,

Thanks for your reply. It's just the java auto instrumentation that I use. I think it's working fine, so it's mainly a UI issue, but I'm also wondering if this could be the intended behavior.

The backend call looks like this:

As you can see, the span types are visible. When I click the 'View full trace' button, I get the following result.

Whereas the bottom part of the trace is still the same.

[Sylvain_Juge]

That definitely looks like a bug to me, I'll ping someone working on UI to see what could be wrong here and what should be the next steps to solve this.

[Harm]

Thanks!

[dgieselaar (Dario Gieselaar)]

Hi @Harm, breakdown data is specific to a service. That means that the service that you are looking at needs to have recorded its own breakdown data. From what I can tell, you are looking at a service that is instrumented by the RUM agent, is that correct? If so, have you enabled breakdownMetrics for that service? See https://www.elastic.co/guide/en/apm/agent/rum-js/current/configuration.html#breakdown-metrics.

[Harm]

Hi @dgieselaar, you're triggering me with your remark saying that spans are specific for a service. The Full Trace combines two different services in this situation. Could it be that it only shows the span types for the frontend service?

[dgieselaar (Dario Gieselaar)]

"Time spent by span type" only shows data recorded by the service that you are looking at. So, if your Java service has breakdown metrics, but the frontend service that is calling it doesn't, the breakdown metrics for transactions happening in the frontend will not display anything in that chart.

In your first screenshot, you're looking at the breakdown metrics for a java service. When you click on "view full trace", you will be looking at breakdown metrics for the service where the trace started, which looks like a frontend (RUM) service. That service might not have breakdown data because the agent for frontend services (the RUM agent) needs to opt-in to report breakdown metrics. In the link I posted earlier there's a description on how you can opt-in: https://www.elastic.co/guide/en/apm/agent/rum-js/current/configuration.html#breakdown-metrics

What happens when you enable that setting as well - assuming that was not the case before?