Tuning APM setup settings

#1

Hi there!
after a success testing and poc of elastic APM in many components and technologies, we are now testing it integrated with all components and thinking what should be decent settings for agents to have good performance of the apm components but enough data for troubleshooting as well.

Firsts tests seems quite heavy because we generated too much data (several GBs for a couple of hours) when we expected to have 10 at most for a day.

So I will start tuning the agent settings as the defaults seem too much but I'd like to ask to community first what is the configuration you have that is valid to you?

  • transaction_sample_rate: default is 1.0. I reduced initially to 0.5 because I thought 100% was too much, still should be reduced? 10% even 5% should be significant enough, what you think?

  • transaction_max_spans

  • span_frames_min_duration: default is 5ms I haven't touched it but seems is commented by default so not applying, I'll play around with it

So, any recommendation on good values for performance but keeping a good instrumentation as well?

Thanks!

#2

Which agent are you using? Most of the agents have a dedicated performance tuning documentation page, see for instance the Python agent: https://www.elastic.co/guide/en/apm/agent/python/current/tuning-and-overhead.html

#3

Additionally you might want to take a look at the APM Server Storage Management giving an overview about storage and sizing and how to reduce storage.

#4

ups sorry didn't comment, java for now.
Thanks I continue reviewing the documentation in detail and trying with different values. My question was very generic just to know if anyone has already tried different values for configuration properties and found a good setting that works on production with high load
cheers

#5

There's not one size fits all answer for the sampling rate. For "web-scale" applications you might even go for sampling rates <1%.