# Distributed tracing missing expansion for one service **URL:** \ **Category:** APM\ **Tags:** ruby\ **Created:** [September 27, 2019, 12:56am UTC](https://discuss.elastic.co/t/distributed-tracing-missing-expansion-for-one-service/201329 "2019-09-27T00:56:39Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![dnorth98](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dnorth98/32/54823_2.png) [@dnorth98](https://discuss.elastic.co/u/dnorth98)\ **Post date:** [September 27, 2019, 12:56am UTC](https://discuss.elastic.co/t/distributed-tracing-missing-expansion-for-one-service/201329/1 "2019-09-27T00:56:39Z")
If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text 🙂 **Kibana version** : 7.3.2 **Elasticsearch version** : 7.3.2 **APM Server version** : 7.3.2 **APM Agent language and version** : 2.11.0 (ruby, rails 5.3) **Browser version** : N/A **Original install method (e.g. download page, yum, deb, from source, etc.) and version**: Elastic Cloud **Fresh install or upgraded from other version?** Fresh Install **Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. N/A **Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**: We're trialing the APM solution and it's all working pretty well. However, there's an odd behaviour with distributed tracing that I'm struggling to understand so I'll try and explain it. First the setup 3 services - webapp, "pod" API and "edge" API. All are ruby Rails applications deployed on AWS Elastic beanstalk. All services have the latest ES APM gem installed. When viewing a trace that involves a call to the 'edge API', it's like the APM agent doesn't see that this is part of the larger trace. For example ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/5/95107e5cbe778881a2af0fbed646e2128e2fdc84.png) In this example above, the trace originates in the webapp. the webapp then calls into the "pod API" (pd-sh-cb-p01-api) and is successfully "matched" and the method from the pod API is shown. However, the next call into the "edge API" (GET api) just shows as a regular http GET with no further information. It turns out this calls a controller in this edge API with a #show method and I verified that when I look in APM at the edge API, that is listed as a transaction. Here's what I see for the call to the pod API (GET pd-sh-cb-p01 in the above shot): ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/1/e178c7abdba090ceaf263cabc132199d27523c35.png) But the call to our "edge API" (GET api..) only shows ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/a/daa6809133e15888bd67a5f827dd469ff8dead01.png) The call is using net-http which is a supported library so I'm at a loss to explain why some calls are being "distributed traced" and others are not. It does seem like it's only calls to (what we call) this edge API so is there some extra config I need to enable here that wouldn't be on the other agents? Spent a few hours trying to see differences but can't come up with any Help appreciated! Cheers ---
**Author:** ![mikker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mikker/32/22118_2.png) [@mikker](https://discuss.elastic.co/u/mikker)\ **Post date:** [September 27, 2019, 12:57pm UTC](https://discuss.elastic.co/t/distributed-tracing-missing-expansion-for-one-service/201329/2 "2019-09-27T12:57:43Z")
Hi Dave! This looks mysterious, you're right. Distributed Tracing works by passing along a header called `Elastic-Apm-Traceparent` with the relevant trace ids. Perhaps check if this is present through all the services? We'll look into this early next week and see if I can figure something out or reproduce it. Mikkel, Elastic ---
**Author:** ![dnorth98](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dnorth98/32/54823_2.png) [@dnorth98](https://discuss.elastic.co/u/dnorth98)\ **Post date:** [September 27, 2019, 7:31pm UTC](https://discuss.elastic.co/t/distributed-tracing-missing-expansion-for-one-service/201329/4 "2019-09-27T19:31:59Z")
Mikkel, I posted that the header wasn't set but I was mistaken. So what I found is in our staging environment, things are working! The big difference is the volume of data in staging is super low (like 1 request per min Vs 50,000/min in prod). Looking in the agent log on the "edge api" service I'm seeing: `Queue is full (256 items), skipping` so I'm guessing we're just missing getting data on the APM server. I also know that the APM server is under-sized while we're on the trial. Reading this: [https://www.elastic.co/guide/en/apm/agent/ruby/current/debugging.html](https://www.elastic.co/guide/en/apm/agent/ruby/current/debugging.html) it looks like there's a few suggestions. What's the best one you've seen work though to keep distributed tracing working? I'd imagine that setting `transaction_sample_rate` to a lower value would cause traces to be missed and we'd have the same problem? Regardless, given that it works in staging, I think this is the root cause....can't view a span that isn't there 🙂 Looking forward to any recommendations you may have. Cheers Dave ---
**Author:** ![dnorth98](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dnorth98/32/54823_2.png) [@dnorth98](https://discuss.elastic.co/u/dnorth98)\ **Post date:** [September 27, 2019, 7:44pm UTC](https://discuss.elastic.co/t/distributed-tracing-missing-expansion-for-one-service/201329/5 "2019-09-27T19:44:07Z")
Ah looks like the server just can't handle the volume too. On the client logs as well as `Queue is full (256 items), skipping` I'm also seeing `APM Server responded with an error: "{\"accepted\":30,\"errors\":[{\"message\":\"queue is full\"}]}\n"` So looks like this is just a pure volume thing. I'll play with some of the client settings (can't change the server size during trial alas) but I suspect there won't be much to do without a bigger server. ---
**Author:** ![mikker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mikker/32/22118_2.png) [@mikker](https://discuss.elastic.co/u/mikker)\ **Post date:** [September 30, 2019, 7:37am UTC](https://discuss.elastic.co/t/distributed-tracing-missing-expansion-for-one-service/201329/6 "2019-09-30T07:37:28Z")
Good to hear you found the cause! Sampling is a good place to start, as you can quickly scale down the payload size and count and then scale it up little by little if needed. Distributed Tracing will still work, as the agent sets the header even though the transaction isn't sampled. The _parent_ will just be the topmost transaction instead of the span. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 21, 2019, 3:37am UTC](https://discuss.elastic.co/t/distributed-tracing-missing-expansion-for-one-service/201329/7 "2019-10-21T03:37:40Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.