Doubt about distributed tracing between services and so much spans

[adrian-arapiles (Adrián)]

I'm using ELK stack to ingest and visualize logs until now.
Now I'm trying add APM to monitorize applications too and I found and casuistic that I don't know if config able to handle.

I have 3 java services (spring). One of them is entry point to generate jobs through http POST requests. When this jobs got executed, generate so many kafka events (from 1k to 500k). Both of others applications (I gonna referenced as client application) listen this kafka topics and consume this events.

With few config, I configure APM and all works very well, including distributed tracing. But when I see transaction of client application, Kibana shows me this warning: Number of items in this trace exceed what is displayed, and I suspect that I'm losing any spans that I can't see. Because there isn't any error log but some transaction has all spans and other transactions hasn't it.

If I click and see parent transaction, I see the trace of http POST request of job generated by entry point application and show whole spans (kafka send events spans and processing of event on client application) until limit (by default 1000) spans is reached.

So, my question is, can I limit the distribute tracing from my entry point application and client application? My intention is that any kafka event to be a transaction in client application and not as span of parent transaction (http request on entry point application).

I haven't instrumented the code.
My agent config is:

service_version=<tag of version>
service_name=<service-name>
hostname=<host-name>
environment=<environment>
application_packages=<my.package>
server_urls=<apm-server-url:9200>
enable_log_correlation=true

If you need any more concise info, please tell me.

Thanks in advance.

Adrián.

[Eyal_Koren]

Hi and thanks for the question.

@adrian-arapiles

So, my question is, can I limit the distribute tracing from my entry point application and client application? My intention is that any kafka event to be a transaction in client application and not as span of parent transaction (http request on entry point application).

Not sure what you mean. What would you like to limit?
Just a clarification - the Kafka record send actions should be traced as a spans, but the receive actions are captured as transactions. I don't think any change in that would assist with what you need because the UI limitation on how many items it can present in one trace will remain the same.

[adrian-arapiles (Adrián)]

First, thanks for your fast reply.

I would try to explain what I expect because maybe I'm wrong.

When I see a trace of receive kafka event and try to see trace logs (I follow this guide about log correlation successfully ) , Kibana shows me logs of whole parent trace, so if there are 500k of kafka events, show me logs of 500k of transactions. And I want to see only the logs referred to current transaction and not parent transaction.

EDIT: Maybe I expect to see log of transaction and not of whole trace. And I have misunderstood.

Sorry about my confusion, maybe I'm using kibana on wrong way.

Thanks.
Adrián.

[Eyal_Koren]

Thanks for your input.
That's actually a very good point, since the transaction ID is logged as well when log correlation is enabled, it may be quite simple to add Transaction logs (in addition to Trace logs) to the Actions menu. If I am not mistaken, it only means using this ID for filtering instead of the trace ID.
We'll take a look.