Java APM agent sample rate ignored

Elastic Observability APM
1

Hi, we are running Elastic APM Agent 1.17.0 on Java 1.8.0_92 Centos 6.5 on several applications.
We are setting sample rate via Kibana, agent receives settings properly

INFO co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Received new configuration from APM Server: {capture_body=off, transaction_max_spans=30, transaction_sample_rate=0.01}

But we do not see any significant changes in Kibana APM graphs, we changed sample rate to 1.0, 0.1, etc, nothing, the app transaction graph keeps at barely the same level.
What could be the reason of this?
This happens only with 1 app, all other are working fine, settings are same
thx.

2

Hi and welcome to the forum!

Not sure what you mean by "frame rate".
The transaction_sample_rate config is about reducing overhead by not sending and storing spans, but the agent still reports time and result for 100% of the transactions.

What effect of this config do you expect to see in the UI that you cannot see?

3

Yeah, mistyped, I meant sample rate.
we have an app, in runs on cluster of 10 VMs
If I set sample rate to 1.0 Kibana shows 80k RPMs, if I set sample rate to 0.01 Kibana shows ~50k RPMs, if I set frame rate to 0.001 Kibana shows the same 50k RPMs

4

The RPM difference you observed is not related to the sample rate.
Transaction metrics like RPM and duration are based on 100% of the transactions, regardless of the sample rate. This setting defines the percentage of transactions for which lower level details will be collected, such as spans, request/response payload etc.

5

However, why do I see difference in RPM? this is not connected to app load, as the difference is only if I set sample rate

6

another thing is that I can see all the messages in ES, it correlates to RPMs.
If I have 80k messages (when I set sample rate to 1.0) - I get 80k RPMs
if I have 40-50k messages (when I set sample rate to 0.1 or 0.001) I get the same 40-50K RPMs
Also we drop unsampled messages on APM Server side with such config
processors:
- drop_event:
when:
equals:
transaction.sampled: false

7

Ahh, if you drop events then RPM would be affected. I can't say why you get more than 1/2 of the transactions reported when setting sample rate to 1% or 0.1% though...

Try setting the java agent log_level to TRACE (to print all sent transaction events to the log) and use much lower loads (100 per minute or so). If you set sample rate to 0.1, then transaction.sampled should be true only for 10% of the transactions (randomly selected).

See if you can provide more info - does this happen only on high loads? Does the agent fail to sample properly? Does the processor fails to delete all unsampled?

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.