APM agent reporting unusually high number of transactions


APM agent is reporting lakhs of transactions for my service whereas we know the QPS for our service is around 1000 (across apis). This was with 100% sampling

I have reduced the sampling to 1% with all body capturing off and max spans as 500. Even with this, APM is reporting requests per minute as lakhs and in a spiky manner which is not true if I look at actual metrics of my service.

I have integrated APM with multiple other services of mine, did not face any issues there.

One difference which I can see in the resource layer of this service which is suffering from incorrect reporting is we are using Google Guice "Providers" in the class and getting an object from it for every API call and proceeding.

One interesting thing to note here is , the sampled transactions for the APIs with Provider injected to do hangle API call does not have any spans within them, even though there are DB calls and network calls from framework which are supported by APM (Hibernate and Jersey, for which I am getting proper spans in other services )

I am stumped by why so high transaction reporting even though when we know for a fact that RPS on the service is no way close to lakhs, its in the region of 1k (aggregated across all APIs).

Also you can see from the data, that APM is reporting top 4 apis who are reporting very high TPS, API response times are around 2ms, whereas in reality its around 700-800ms

Are you aware of any existing issues or bugs in APM Agent which leads to this?

PFA Screen Shot

Looking at legend of the RPM graph, I see that the throughput of 2xx is around 60k rpm. This seems to be in line with the 1k rps you observed.

60k rpm is the average in that timeframe. But as the graph shows, the throughput seems to be quite spikey. So in some time windows, the throughput is higher than the average.

Could it be that the actual response time is 700-800µs (microseconds) and that the UI rounds that to 1ms (milleseconds)?

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.