# How to drop spans with duration below 5ms **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [June 20, 2019, 7:34am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617 "2019-06-20T07:34:07Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Askia\_Mohamed\_Kadri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askia_mohamed_kadri/32/134665_2.png) [@Askia\_Mohamed\_Kadri](https://discuss.elastic.co/u/Askia_Mohamed_Kadri)\ **Post date:** [June 20, 2019, 7:34am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/1 "2019-06-20T07:34:07Z")
**Kibana version** : 6.8 **Elasticsearch version** : 6.8 **APM Server version** : 6.8 **APM Agent language and version** : JavaAgent 1.7 **Browser version** : **Original install method (e.g. download page, yum, deb, from source, etc.) and version**: download tar.gz from elastic official site I'm using JavaAgent to instument my app. I made a dev that uses the API to add @CaptureSpan on all the compiled classes from a specific package of the jar so that my app methods can be reported as span. Since then everything is alright. My app methods are well reported. But as I'm restricted to 500 spans (I dont' want to increase it to avoid overhead) I would like to drop all methods below a certain duration. **Can somebody please tell me how to do this?** I tried setting `trace_methods_duration_threshold` but it's not working, even using `trace_methods` ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 20, 2019, 8:10am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/2 "2019-06-20T08:10:03Z")
Hi and thanks for the question. The `trace_methods_duration_threshold` takes effect only on methods instrumented through `trace_methods`, but since you added `@CaptureSpan` to everything, it has no effect. You should remove the `@CaptureSpan` and instead configure the `trace_methods` to trace this entire package. I hope this helps, Eyal. ---
**Author:** ![Askia\_Mohamed\_Kadri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askia_mohamed_kadri/32/134665_2.png) [@Askia\_Mohamed\_Kadri](https://discuss.elastic.co/u/Askia_Mohamed_Kadri)\ **Post date:** [June 20, 2019, 8:13am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/3 "2019-06-20T08:13:24Z")
Hi @Eyal_Koren, thanks for your quick reply. I'm going to try this. I'll let you know if it worked for me. Thanks again. ---
**Author:** ![Askia\_Mohamed\_Kadri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askia_mohamed_kadri/32/134665_2.png) [@Askia\_Mohamed\_Kadri](https://discuss.elastic.co/u/Askia_Mohamed_Kadri)\ **Post date:** [June 27, 2019, 7:49am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/4 "2019-06-27T07:49:03Z")
Hi @Eyal_Koren, I'm back a week after for some new questions. I used the `trace_methods` and the `trace_methods_duration_threshold`but still not having the result I'm waiting for. I'll explain you maybe you can tell me what I'm missing. Let's say my `application_packages` is `com.example.kimchy`, and my `trace_methods` is `com.example.kimchy.service.*#*` as the methods I really want to instrument are there. To drop spans not taking longer than 100ms I set`trace_methods_duration_threshold` to 100ms. I had not that much but some spans reprensented in APM UI that guide me to understand that I was loosing time in methods related to activiti. So I decided to add org.activiti in `application_packages` and org.activiti.\* in `trace_methods`. But I'm not having such more results. Many span are not represented in APM UI and many methods are not present in spans. What I'm I missing? How can I have all spans represented in APM UI and how to have all methods from my packages? Are they some methods dropped automatically or to avoid intrusion APM doens't really instument all methods? ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 27, 2019, 8:19am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/5 "2019-06-27T08:19:48Z")
Most methods are expected to be much faster than 100ms. What happens if you set the threshold to 1ms? ---
**Author:** ![Askia\_Mohamed\_Kadri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askia_mohamed_kadri/32/134665_2.png) [@Askia\_Mohamed\_Kadri](https://discuss.elastic.co/u/Askia_Mohamed_Kadri)\ **Post date:** [June 27, 2019, 8:22am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/6 "2019-06-27T08:22:38Z")
In fact yes, but even with the filter I'm getting span with duration less than 100 micro seconds for example. And the methods I'd like to see are methods with duration over 30s for example. But I can't see what methods have been executed after ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 27, 2019, 8:26am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/7 "2019-06-27T08:26:51Z")
Please send the entire configuration you are using and screenshots showing what doesn't work properly ---
**Author:** ![Askia\_Mohamed\_Kadri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askia_mohamed_kadri/32/134665_2.png) [@Askia\_Mohamed\_Kadri](https://discuss.elastic.co/u/Askia_Mohamed_Kadri)\ **Post date:** [June 27, 2019, 8:27am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/8 "2019-06-27T08:27:29Z")
Here is a typical example. I really would like to see what happens next. All the methods executed after are part of my packages I asked to be instrumented. I'm not getting more than 500 spans (even though I increased tho max number of spans) ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/e/4e84a11c2b51db0e6088c6bf917acad81d50d26e.png) ---
**Author:** ![Askia\_Mohamed\_Kadri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askia_mohamed_kadri/32/134665_2.png) [@Askia\_Mohamed\_Kadri](https://discuss.elastic.co/u/Askia_Mohamed_Kadri)\ **Post date:** [June 27, 2019, 8:32am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/9 "2019-06-27T08:32:49Z")
Here is the link to a pastebin with my elasticapm.properties [https://pastebin.com/ENb7SyWL](https://pastebin.com/ENb7SyWL) ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 27, 2019, 8:37am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/10 "2019-06-27T08:37:55Z")
This is from [the `settrace_methods_duration_threshold ` documentation](https://www.elastic.co/guide/en/apm/agent/java/current/config-core.html#config-trace-methods-duration-threshold): > Such methods will be traced and reported if one of the following applies: > > - This method’s duration crossed the configured threshold. > - This method ended with Exception. > - A method executed as part of the execution of this method crossed the threshold or ended with Exception. > - A "forcibly-traced method" (e.g. DB queries, HTTP exits, custom) was executed during the execution of this method. Due to the way the agent works, we must keep complete traces, so we must send all methods leading to a method we want to keep. We can discard only spans on branches that do not directly lead to forcibly traced spans. The way to reduce this and make sure it makes more sense it to be more specific with the `trace_methods` configuration. This configuration was introduced in order to trace specific methods. In order to get what you want, you will need to spend some time on it and be very specific in what you want to get spans for. ---
**Author:** ![Askia\_Mohamed\_Kadri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askia_mohamed_kadri/32/134665_2.png) [@Askia\_Mohamed\_Kadri](https://discuss.elastic.co/u/Askia_Mohamed_Kadri)\ **Post date:** [June 27, 2019, 9:06am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/11 "2019-06-27T09:06:15Z")
I understand what you mean. I'm going to add specific methods to see what I'll. But still don't have the response on why I'm not having all my spans for a transaction represented on the APM UI even if I got them in the spans from the transaction spans dashboard ---
**Author:** ![Askia\_Mohamed\_Kadri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askia_mohamed_kadri/32/134665_2.png) [@Askia\_Mohamed\_Kadri](https://discuss.elastic.co/u/Askia_Mohamed_Kadri)\ **Post date:** [June 27, 2019, 9:14am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/12 "2019-06-27T09:14:39Z")
This is an example. I have here a transaction that took 200s. But no spans are represented. ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/d/ed16e8f0c48459b43eeee448d58c0c95ad8e0767.png) When I dive into transaction sample documents to see spans I can see that there are 2681 spans generated. ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/d/0d20ff7d5a64b80491804327be4a96ceac6dc9c5.png) How can I solve this? Is there a limit of represented spans? It doesn't occur on all transactions. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 27, 2019, 9:29am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/13 "2019-06-27T09:29:32Z")
> [@Askia\_Mohamed\_Kadri](#): > > Is there a limit of represented spans? Yes, I think the UI will not allow getting more than 1000 spans per view, so if there are missing, it can't draw the tree. You set `transaction_max_spans=2000`, which really you shouldn't (I think you are getting more than 2000 because there are async operations where we don't enforce this limit). This way of using the `trace_methods` configuration is unrecommended. Spans should represent meaningful events, not just any method invocation. Not only this will make your waterfall view difficult to read, it is also very un-scalable. In the future, we may add the ability to show call trees based on statistical/sampling profilers, which are more fit in terms of overhead. ---
**Author:** ![Askia\_Mohamed\_Kadri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askia_mohamed_kadri/32/134665_2.png) [@Askia\_Mohamed\_Kadri](https://discuss.elastic.co/u/Askia_Mohamed_Kadri)\ **Post date:** [June 27, 2019, 9:50am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/14 "2019-06-27T09:50:50Z")
Alright, I understand now. It would be nice to able to represented a certain number of spans too only based on a duration filtering ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [June 27, 2019, 10:25am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/15 "2019-06-27T10:25:26Z")
Would be super nice, but requires a big rework... Again- this feature is meant for something else, which is viewing detailed info on significant events ---
**Author:** ![Askia\_Mohamed\_Kadri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/askia_mohamed_kadri/32/134665_2.png) [@Askia\_Mohamed\_Kadri](https://discuss.elastic.co/u/Askia_Mohamed_Kadri)\ **Post date:** [June 28, 2019, 9:32am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/16 "2019-06-28T09:32:11Z")
Yes, I understand what you means. Thanks a lot for all your responses. Best regards ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 19, 2019, 5:33am UTC](https://discuss.elastic.co/t/how-to-drop-spans-with-duration-below-5ms/186617/17 "2019-07-19T05:33:53Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.