# Doubt about distributed tracing between services and so much spans

**URL:** <https://discuss.elastic.co/t/doubt-about-distributed-tracing-between-services-and-so-much-spans/247870>\
**Category:** APM\
**Tags:** java, ui\
**Created:** [September 8, 2020, 10:21am UTC](https://discuss.elastic.co/t/doubt-about-distributed-tracing-between-services-and-so-much-spans/247870 "2020-09-08T10:21:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![adrian-arapiles](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrian-arapiles/32/101173_2.png) [@adrian-arapiles](https://discuss.elastic.co/u/adrian-arapiles)\
**Post date:** [September 8, 2020, 10:21am UTC](https://discuss.elastic.co/t/doubt-about-distributed-tracing-between-services-and-so-much-spans/247870/1 "2020-09-08T10:21:37Z")

</div>

I'm using ELK stack to ingest and visualize logs until now.  
Now I'm trying add APM to monitorize applications too and I found and casuistic that I don't know if config able to handle.

I have 3 java services (spring). One of them is entry point to generate jobs through http POST requests. When this jobs got executed, generate so many kafka events (from 1k to 500k). Both of others applications (I gonna referenced as _client application_) listen this kafka topics and consume this events.

With few config, I configure APM and all works very well, including distributed tracing. But when I see transaction of client application, Kibana shows me this warning: **Number of items in this trace exceed what is displayed** , and I suspect that I'm losing any spans that I can't see. Because there isn't any error log but some transaction has all spans and other transactions hasn't it.

If I click and see parent transaction, I see the trace of http POST request of job generated by entry point application and show whole spans (kafka send events spans and processing of event on client application) until limit (by default 1000) spans is reached.

So, my question is, can I limit the distribute tracing from my entry point application and client application? My intention is that any kafka event to be a transaction in client application and not as span of parent transaction (http request on entry point application).

I haven't instrumented the code.  
My agent config is:

```auto
service_version=<tag of version>
service_name=<service-name>
hostname=<host-name>
environment=<environment>
application_packages=<my.package>
server_urls=<apm-server-url:9200>
enable_log_correlation=true

```

If you need any more concise info, please tell me.

Thanks in advance.

Adrián.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [September 8, 2020, 10:38am UTC](https://discuss.elastic.co/t/doubt-about-distributed-tracing-between-services-and-so-much-spans/247870/2 "2020-09-08T10:38:53Z")

</div>

Hi and thanks for the question.

> [@adrian-arapiles](#):
>
> So, my question is, can I limit the distribute tracing from my entry point application and client application? My intention is that any kafka event to be a transaction in client application and not as span of parent transaction (http request on entry point application).

Not sure what you mean. What would you like to limit?  
Just a clarification - the Kafka record `send` actions should be traced as a spans, but the `receive` actions are captured as transactions. I don't think any change in that would assist with what you need because the UI limitation on how many items it can present in one trace will remain the same.

---

<div class="post-metadata">

**Author:** ![adrian-arapiles](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrian-arapiles/32/101173_2.png) [@adrian-arapiles](https://discuss.elastic.co/u/adrian-arapiles)\
**Post date:** [September 8, 2020, 11:33am UTC](https://discuss.elastic.co/t/doubt-about-distributed-tracing-between-services-and-so-much-spans/247870/3 "2020-09-08T11:33:37Z")

</div>

First, thanks for your fast reply.

I would try to explain what I expect because maybe I'm wrong.

When I see a trace of `receive` kafka event and try to see trace logs (I follow this [guide](https://www.elastic.co/guide/en/apm/agent/java/current/log-correlation.html) about log correlation successfully 🎉 ) , Kibana shows me logs of whole parent trace, so if there are 500k of kafka events, show me logs of 500k of transactions. And I want to see only the logs referred to current transaction and not parent transaction.

EDIT: Maybe I expect to see log of transaction and not of whole trace. And I have misunderstood.

Sorry about my confusion, maybe I'm using kibana on wrong way.

Thanks.  
Adrián.

---

<div class="post-metadata">

**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\
**Post date:** [September 9, 2020, 1:15pm UTC](https://discuss.elastic.co/t/doubt-about-distributed-tracing-between-services-and-so-much-spans/247870/4 "2020-09-09T13:15:35Z")

</div>

Thanks for your input.  
That's actually a very good point, since the transaction ID is logged as well when log correlation is enabled, it may be quite simple to add `Transaction logs` (in addition to `Trace logs`) to the `Actions` menu. If I am not mistaken, it only means using this ID for filtering instead of the trace ID.  
We'll take a look.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 30, 2020, 9:15am UTC](https://discuss.elastic.co/t/doubt-about-distributed-tracing-between-services-and-so-much-spans/247870/5 "2020-09-30T09:15:51Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.
