Elastic APM using Java agent : Only a subset of my incoming requests have spans, while the majority of requests do not

[Hanok_Jacob]

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version:7.9.2

Elasticsearch version: 7.9.2

APM Server version: 7.9.2

APM Agent language and version: java agent, version : 1.42.0

Browser version: Google chrom, Version 117.0.5938.92 (Official Build) (arm64)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Downloaded

Fresh install or upgraded from other version?
Fresh install

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
Nothing special about the setup. It is the standard setup as mentioned in the doc.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I have a Spring MVC application running on Google AppEngine with Java 17 as the runtime. I've configured it with the -javaagent flag. My goal is to monitor and view all the requests and their associated spans in my application. However, I've encountered an issue where I can only see spans for some of the incoming requests. While I can see the overall execution time when filtering by trace IDs, I'm unable to view the specific spans related to those requests. I need assistance in understanding why this is happening and what steps I should take to ensure that I can view spans for all incoming requests.

The default setting for request tracing is to capture all incoming requests. However, I've explicitly set the transaction_sample_rate to 1 to ensure that all requests are sampled for tracing. Surprisingly, even with this configuration, I'm still unable to capture traces for all requests.

These are the configs that I have used:
-javaagent:WEB-INF/lib/elastic-apm-agent-1.42.0.jar -Delastic.apm.service_name=myApp -Delastic.apm.environment=staging -Delastic.apm.server_urls=http://xx.yy.zz.aa:8200 -Delastic.apm.secret_token=******* -Delastic.apm.application_packages=com.hanok -Delastic.apm.trace_methods=com.hanok.*

This is the expected result where I could see all the spans:

Steps to reproduce:
1.Deploy the app using Java agent configs
2.Try filtering out using a trace.id
3.This is what I get as the result which does not have spans at all :

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

[Jack_Shirazi]

Is it just the first trace that is missing the child spans?

[Hanok_Jacob]

The majority of the requests are missing the child spans. For eg: if I trigger 100 GET requests, I can only see the whole traces for a couple of the requests (10-15 requests).

[Jack_Shirazi]

You can set the agent log to DEBUG and it lists all the transactions and spans that the agent is generating. That will let you determine if the agent is not producing the child spans or whether the spans are being produced but not showing in the UI. That would be the next step

[Hanok_Jacob]

Can you please tell me what I should be looking for? After setting elastic.apm.log_level= DEBUG
It did not show any errors. These are my findings :
The request that has all the spans has this log : 2023-09-28 12:12:45,146 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 21).
The request that does not have any spans does not have this log.

[Jack_Shirazi]

That log is not relevant, it's the comms thread. What you want to look for are the actual spans and transactions, for example search for 'endSpan' and 'endTransaction' to find trace IDs and then you can use the trace ID to see all the spans in a particular transaction in the log, and correlate that with what you see in the UI - which also has the trace ID

[Hanok_Jacob]

I know the trace IDs @Jack_Shirazi. Maybe I confused you. I don't have a problem finding the trace IDs. When I use the trace IDs to filter it only gives me the overview of the transaction. The child spans are missing. Some requests have all the spans and some don’t as I have mentioned previously. I want to know why the spans are missing for the majority of my requests.

This is the trace ID that I got from the logs (highlighted in blue)

[Jack_Shirazi]

Are the child spans missing in the log though? The point is to first determine if this is a UI problem or the spans are not generated by the agent

[Hanok_Jacob]

I could see start and end spans from the debug logs but I checked the span index using trace.id query in Kibana. which also resulted in 0 hits. I don't think it's a UI problem.

[Jack_Shirazi]

Sorry I wasn't clear. Choose two traceIDs from the UI, one that has the spans and one that doesn't. Then filter the logs for those traceIDs. You should see exactly the same spans and transactions as the UI if it's on the agent side. If the traceID which has no spans in the UI has spans in the logs, then it's UI. This is definitive one way or the other, there is no need to infer anything

If it's missing the spans in the logs as well as the UI, then either the the methods that would generate the spans don't get called for those requests, or the agent is choosing to not generate those spans for some reason. The latter could be eg from a span_min_duration setting