High CPU usage with Python agent

Elastic Observability APM
1

Hello!

I have a webserver running a Django application and have started to use the Elastic APM since Opbeat was shut down. I'm currently only doing instrumentation and I'm experiencing high CPU loads. I used Opbeat before migrating to Elastic APM without experiencing CPU loads as high as with Elastic APM.

The only configuration I've done is to setup SECRET_TOKEN, SERVER_URL, SERVICE_VERSION and SERVICE_NAME in the ELASTIC_APM setting in Django settings.

I know about the transaction sample rate setting and it would presumably lower the CPU load but I wouldn't expect Elastic APM to perform that different from Opbeat.

Do you have any suggestions to what I might have misconfigured or can you help me debug this issue in any way?

2

Hi David

Thanks for raising this issue! In some cases, the Python agent can indeed use more CPU than the Opbeat agent. This is because the Opbeat agent did some aggressive optimization and caching that was later resolved in the Opbeat backend. Due to how Elastic APM works, these optimizations are no longer possible.

We benchmarked the Python agent a lot, and the largest hot spot tends to be collection of stack frames for spans. If your code generates a lot of short spans, one option is to disable stack frame collection for very short spans using the span_frames_min_duration setting, e.g. set it to 5 to not collect any span stack frames for spans shorter than 5 milliseconds.

Also have a look at our Tuning and Overhead docs page if you haven't seen it yet.

3

Yeah there's a large amount of short spans. Now with a span duration limit of 5ms the CPU-load is lower. I will try even higher limits and probably set the transaction sample rate as well. Thanks for your help!

4

Good to hear it has an effect! Please keep us up to date on your findings, as this is a subject that is very important to us. We're e.g. considering to setting a default value for span_frames_min_duration, so it would definitely be interesting to hear how and on which value you settle!

5

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.