# Elastic APM using Java agent : Only a subset of my incoming requests have spans, while the majority of requests do not **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [September 26, 2023, 12:10pm UTC](https://discuss.elastic.co/t/elastic-apm-using-java-agent-only-a-subset-of-my-incoming-requests-have-spans-while-the-majority-of-requests-do-not/343872 "2023-09-26T12:10:04Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Hanok\_Jacob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hanok_jacob/32/126013_2.png) [@Hanok\_Jacob](https://discuss.elastic.co/u/Hanok_Jacob)\ **Post date:** [September 26, 2023, 12:10pm UTC](https://discuss.elastic.co/t/elastic-apm-using-java-agent-only-a-subset-of-my-incoming-requests-have-spans-while-the-majority-of-requests-do-not/343872/1 "2023-09-26T12:10:04Z")
If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text 🙂 TIP 1: select at least one tag that further categorizes your topic. For example `server` for APM Server related questions, `java` for questions regarding the Elastic APM Java agent, or `ui` for questions about the APM App within Kibana. TIP 2: Check out the [troubleshooting guide](https://www.elastic.co/guide/en/apm/get-started/current/troubleshooting.html) first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you. **Kibana version** :7.9.2 **Elasticsearch version** : 7.9.2 **APM Server version** : 7.9.2 **APM Agent language and version** : java agent, version : 1.42.0 **Browser version** : Google chrom, Version 117.0.5938.92 (Official Build) (arm64) **Original install method (e.g. download page, yum, deb, from source, etc.) and version**: Downloaded **Fresh install or upgraded from other version?** Fresh install **Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. Nothing special about the setup. It is the standard setup as mentioned in the doc. **Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**: I have a Spring MVC application running on Google AppEngine with Java 17 as the runtime. I've configured it with the `-javaagent` flag. My goal is to monitor and view all the requests and their associated spans in my application. However, I've encountered an issue where I can only see spans for some of the incoming requests. While I can see the overall execution time when filtering by trace IDs, I'm unable to view the specific spans related to those requests. I need assistance in understanding why this is happening and what steps I should take to ensure that I can view spans for all incoming requests. The default setting for request tracing is to capture all incoming requests. However, I've explicitly set the `transaction_sample_rate` to 1 to ensure that all requests are sampled for tracing. Surprisingly, even with this configuration, I'm still unable to capture traces for all requests. These are the configs that I have used: -javaagent:WEB-INF/lib/elastic-apm-agent-1.42.0.jar -Delastic.apm.service\_name=myApp -Delastic.apm.environment=staging -Delastic.apm.server\_urls=[http://xx.yy.zz.aa:8200](http://xx.yy.zz.aa:8200) -Delastic.apm.secret\_token=\*\*\*\*\*\*\* -Delastic.apm.application\_packages=com.hanok -Delastic.apm.trace\_methods=com.hanok.\* This is the expected result where I could see all the spans: ![Screenshot 2023-09-26 at 4.02.11 PM](https://us1.discourse-cdn.com/elastic/original/3X/8/e/8e66f9d4829a75dcf0136daf96574953ff2ae277.png) **Steps to reproduce** : 1.Deploy the app using Java agent configs 2.Try filtering out using a trace.id 3.This is what I get as the result which does not have spans at all : ![Screenshot 2023-09-26 at 4.03.15 PM](https://us1.discourse-cdn.com/elastic/original/3X/e/4/e442ccc01230997982135f28953d4891e0f30d58.png) **Errors in browser console (if relevant)**: **Provide logs and/or server output (if relevant)**: ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [September 28, 2023, 10:45am UTC](https://discuss.elastic.co/t/elastic-apm-using-java-agent-only-a-subset-of-my-incoming-requests-have-spans-while-the-majority-of-requests-do-not/343872/2 "2023-09-28T10:45:29Z")
Is it just the first trace that is missing the child spans? ---
**Author:** ![Hanok\_Jacob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hanok_jacob/32/126013_2.png) [@Hanok\_Jacob](https://discuss.elastic.co/u/Hanok_Jacob)\ **Post date:** [September 28, 2023, 10:49am UTC](https://discuss.elastic.co/t/elastic-apm-using-java-agent-only-a-subset-of-my-incoming-requests-have-spans-while-the-majority-of-requests-do-not/343872/3 "2023-09-28T10:49:44Z")
The majority of the requests are missing the child spans. For eg: if I trigger 100 GET requests, I can only see the whole traces for a couple of the requests (10-15 requests). ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [September 28, 2023, 11:16am UTC](https://discuss.elastic.co/t/elastic-apm-using-java-agent-only-a-subset-of-my-incoming-requests-have-spans-while-the-majority-of-requests-do-not/343872/4 "2023-09-28T11:16:03Z")
You can set the agent log to DEBUG and it lists all the transactions and spans that the agent is generating. That will let you determine if the agent is not producing the child spans or whether the spans are being produced but not showing in the UI. That would be the next step ---
**Author:** ![Hanok\_Jacob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hanok_jacob/32/126013_2.png) [@Hanok\_Jacob](https://discuss.elastic.co/u/Hanok_Jacob)\ **Post date:** [September 29, 2023, 5:26am UTC](https://discuss.elastic.co/t/elastic-apm-using-java-agent-only-a-subset-of-my-incoming-requests-have-spans-while-the-majority-of-requests-do-not/343872/5 "2023-09-29T05:26:07Z")
Can you please tell me what I should be looking for? After setting `elastic.apm.log_level= DEBUG ` It did not show any errors. These are my findings : The request that has all the spans has this log : `2023-09-28 12:12:45,146 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 21)`. The request that does not have any spans does not have this log. ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [September 29, 2023, 7:49am UTC](https://discuss.elastic.co/t/elastic-apm-using-java-agent-only-a-subset-of-my-incoming-requests-have-spans-while-the-majority-of-requests-do-not/343872/6 "2023-09-29T07:49:08Z")
That log is not relevant, it's the comms thread. What you want to look for are the actual spans and transactions, for example search for 'endSpan' and 'endTransaction' to find trace IDs and then you can use the trace ID to see all the spans in a particular transaction in the log, and correlate that with what you see in the UI - which also has the trace ID ---
**Author:** ![Hanok\_Jacob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hanok_jacob/32/126013_2.png) [@Hanok\_Jacob](https://discuss.elastic.co/u/Hanok_Jacob)\ **Post date:** [September 29, 2023, 9:42am UTC](https://discuss.elastic.co/t/elastic-apm-using-java-agent-only-a-subset-of-my-incoming-requests-have-spans-while-the-majority-of-requests-do-not/343872/7 "2023-09-29T09:42:45Z")
I know the trace IDs @Jack_Shirazi. Maybe I confused you. I don't have a problem finding the trace IDs. When I use the trace IDs to filter it only gives me the overview of the transaction. The child spans are missing. Some requests have all the spans and some don’t as I have mentioned previously. I want to know why the spans are missing for the majority of my requests. ![Screenshot 2023-09-29 at 3.08.04 PM](https://us1.discourse-cdn.com/elastic/original/3X/f/e/fe1316ac906d17d2004700c3100d64a51177e2b1.png) ![Screenshot 2023-09-29 at 3.09.21 PM](https://us1.discourse-cdn.com/elastic/original/3X/c/e/ce6c5da186a31371aa396cfd95fc52371372a088.png) ![Screenshot 2023-09-29 at 3.09.44 PM](https://us1.discourse-cdn.com/elastic/original/3X/f/0/f0b6c29bf49ef1095329c75582a8b325b600b749.png) This is the trace ID that I got from the logs (highlighted in blue) ![Screenshot 2023-09-29 at 3.22.47 PM](https://us1.discourse-cdn.com/elastic/original/3X/e/0/e0dd8d62051640bb762208314d954b40e3614c7a.png) ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [September 29, 2023, 10:57am UTC](https://discuss.elastic.co/t/elastic-apm-using-java-agent-only-a-subset-of-my-incoming-requests-have-spans-while-the-majority-of-requests-do-not/343872/8 "2023-09-29T10:57:49Z")
Are the child spans missing in the log though? The point is to first determine if this is a UI problem or the spans are not generated by the agent ---
**Author:** ![Hanok\_Jacob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hanok_jacob/32/126013_2.png) [@Hanok\_Jacob](https://discuss.elastic.co/u/Hanok_Jacob)\ **Post date:** [September 29, 2023, 11:11am UTC](https://discuss.elastic.co/t/elastic-apm-using-java-agent-only-a-subset-of-my-incoming-requests-have-spans-while-the-majority-of-requests-do-not/343872/9 "2023-09-29T11:11:47Z")
I could see start and end spans from the debug logs but I checked the span index using trace.id query in Kibana. which also resulted in 0 hits. I don't think it's a UI problem. ![Screenshot 2023-09-29 at 4.42.58 PM](https://us1.discourse-cdn.com/elastic/original/3X/8/5/8595c41f5f5322eaf138b375109b59d45bb7a0c4.png) ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [September 29, 2023, 1:02pm UTC](https://discuss.elastic.co/t/elastic-apm-using-java-agent-only-a-subset-of-my-incoming-requests-have-spans-while-the-majority-of-requests-do-not/343872/10 "2023-09-29T13:02:26Z")
Sorry I wasn't clear. Choose two traceIDs from the UI, one that has the spans and one that doesn't. Then filter the logs for those traceIDs. You should see exactly the same spans and transactions as the UI if it's on the agent side. If the traceID which has no spans in the UI has spans in the logs, then it's UI. This is definitive one way or the other, there is no need to infer anything If it's missing the spans in the logs as well as the UI, then either the the methods that would generate the spans don't get called for those requests, or the agent is choosing to not generate those spans for some reason. The latter could be eg from a [span\_min\_duration](https://www.elastic.co/guide/en/apm/agent/java/current/config-core.html#config-span-min-duration) setting ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 27, 2023, 1:03pm UTC](https://discuss.elastic.co/t/elastic-apm-using-java-agent-only-a-subset-of-my-incoming-requests-have-spans-while-the-majority-of-requests-do-not/343872/11 "2023-10-27T13:03:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.