Choppy graphs with APM 8.x

Elastic Observability APM
,
1

Not sure if this is just a very delayed duplicate of Dropped unsampled transactions in >= 8.x poses a problem - but I’ve only just upgraded to version 8 of the various stack components.

The thoughput/latency graphs shown in Kibana aren’t particularly useful at the moment, as shown in this screenshot:

kibana_vs_grafana
kibana_vs_grafana1240×967 169 KB

The sample rate for this particular app is 0.001 - pretty low, but it’s all we need.

Comparing this to logs parsed from uWSGI through filebeat and shown in Grafana (the second graph) - the APM graph isn’t particularly useful.

Is there any way of getting better metrics for throughput/latency in version 8?

Kibana version: 8.19.9
Elasticsearch version: 8.19.9
APM Server version: 8.19.9
APM Agent language and version: Python/Django, 6.23.0 (various versions)
Fresh install or upgraded from other version? Fresh install

2

Hi @Alex-T ,

I assume this is the Throughput graph in APM UI. My understanding is that because of a low sample rate at 0.1%, and that apm-server 8.x no longer store unsampled transactions, the throughput graph becomes choppy.

In your case, assuming an average 300tpm, at 0.1% sample rate, apm-server receives 0.3 trace per minute. Then, for each transaction apm-server receives, apm-server extrapolates the “representative count” by multiplying this trace event by the inverse of sample rate, e.g. each transaction event represents 1000 transactions.

In short, what you’re seeing in your graph is an unfortunate combination of low sample rate, low throughput, and the cost savings improvement in 8.x.

As for workarounds,

  • apm-server.sampling.keep_unsampled was removed in 8.0 and there is no way to enable it.
  • You may enable 100% sampling on the client and use * Tail-based sampling , so that apm-server sees every transaction event and can produce a less choppy graph, while storing the same sampled number of events in Elasticsearch. This comes with an added benefit as apm-server tail-based sampling has a bias to sample slow / failure traces. However, it requires a fast disk and more CPU power. On a side note, 9.x comes with major performance improvements in TBS.
1 Like