How to drop spans with duration below 5ms

Kibana version: 6.8

Elasticsearch version: 6.8

APM Server version: 6.8

APM Agent language and version: JavaAgent 1.7

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: download tar.gz from elastic official site

I'm using JavaAgent to instument my app.
I made a dev that uses the API to add @CaptureSpan on all the compiled classes from a specific package of the jar so that my app methods can be reported as span. Since then everything is alright. My app methods are well reported.
But as I'm restricted to 500 spans (I dont' want to increase it to avoid overhead) I would like to drop all methods below a certain duration.
Can somebody please tell me how to do this?
I tried setting trace_methods_duration_threshold but it's not working, even using trace_methods

Hi and thanks for the question.

The trace_methods_duration_threshold takes effect only on methods instrumented through trace_methods, but since you added @CaptureSpan to everything, it has no effect.
You should remove the @CaptureSpan and instead configure the trace_methods to trace this entire package.

I hope this helps,

Hi @Eyal_Koren, thanks for your quick reply.
I'm going to try this. I'll let you know if it worked for me.

Thanks again.

Hi @Eyal_Koren,

I'm back a week after for some new questions.
I used the trace_methods and the trace_methods_duration_thresholdbut still not having the result I'm waiting for. I'll explain you maybe you can tell me what I'm missing.
Let's say my application_packages is com.example.kimchy, and my trace_methods is com.example.kimchy.service.*#* as the methods I really want to instrument are there. To drop spans not taking longer than 100ms I settrace_methods_duration_threshold to 100ms.
I had not that much but some spans reprensented in APM UI that guide me to understand that I was loosing time in methods related to activiti.
So I decided to add org.activiti in application_packages and org.activiti.* in trace_methods. But I'm not having such more results.
Many span are not represented in APM UI and many methods are not present in spans.

What I'm I missing?
How can I have all spans represented in APM UI and how to have all methods from my packages?
Are they some methods dropped automatically or to avoid intrusion APM doens't really instument all methods?

Most methods are expected to be much faster than 100ms. What happens if you set the threshold to 1ms?

In fact yes, but even with the filter I'm getting span with duration less than 100 micro seconds for example.
And the methods I'd like to see are methods with duration over 30s for example. But I can't see what methods have been executed after

Please send the entire configuration you are using and screenshots showing what doesn't work properly

Here is a typical example. I really would like to see what happens next. All the methods executed after are part of my packages I asked to be instrumented.
I'm not getting more than 500 spans (even though I increased tho max number of spans)

Here is the link to a pastebin with my

This is from the settrace_methods_duration_threshold documentation:

Such methods will be traced and reported if one of the following applies:

  • This method’s duration crossed the configured threshold.
  • This method ended with Exception.
  • A method executed as part of the execution of this method crossed the threshold or ended with Exception.
  • A "forcibly-traced method" (e.g. DB queries, HTTP exits, custom) was executed during the execution of this method.

Due to the way the agent works, we must keep complete traces, so we must send all methods leading to a method we want to keep. We can discard only spans on branches that do not directly lead to forcibly traced spans.

The way to reduce this and make sure it makes more sense it to be more specific with the trace_methods configuration. This configuration was introduced in order to trace specific methods. In order to get what you want, you will need to spend some time on it and be very specific in what you want to get spans for.

I understand what you mean. I'm going to add specific methods to see what I'll.
But still don't have the response on why I'm not having all my spans for a transaction represented on the APM UI even if I got them in the spans from the transaction spans dashboard

This is an example. I have here a transaction that took 200s. But no spans are represented.

When I dive into transaction sample documents to see spans I can see that there are 2681 spans generated.

How can I solve this?
Is there a limit of represented spans?
It doesn't occur on all transactions.

Yes, I think the UI will not allow getting more than 1000 spans per view, so if there are missing, it can't draw the tree.

You set transaction_max_spans=2000, which really you shouldn't (I think you are getting more than 2000 because there are async operations where we don't enforce this limit). This way of using the trace_methods configuration is unrecommended. Spans should represent meaningful events, not just any method invocation. Not only this will make your waterfall view difficult to read, it is also very un-scalable.

In the future, we may add the ability to show call trees based on statistical/sampling profilers, which are more fit in terms of overhead.

Alright, I understand now. It would be nice to able to represented a certain number of spans too only based on a duration filtering

Would be super nice, but requires a big rework...
Again- this feature is meant for something else, which is viewing detailed info on significant events

Yes, I understand what you means.
Thanks a lot for all your responses.
Best regards

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.