Time for apm data to show up in kibana apm interface

[Peter_Li]

I am wondering what is the expected time for a transaction/span data to show up in kibana ?
I noticed that it is taking 1-2 minutes after the close of transaction before data shows up in kibana.

I am running 8.4.2 (elasticsearch, kibana, apm) and fleet managed apm server.

Are there ways to improve the response speed ? Where to find the bottleneck ?

Thanks

[trentm (Trent Mick)]

Hi @Peter_Li,

I'll answer as best I can. I know the Node.js APM agent well, but my knowledge on APM server and the Kibana APM app are limited.

The first step is for the APM agent to send the events to APM server. Most language APM agents will have some queuing delay, but it depends on which agent you are using. What language/APM agent are you using?

After that, there is some queuing in APM server, but I don't know the details well. In my limited experience APM server writing the trace data to Elasticsearch is quick.

After that, it may depend how you are observing in Kibana when the transaction/span data "shows up". If this is for a query in Discover or something, then there shouldn't be a delay. If it is for a chart that is based on aggregated transaction metrics, then there is some bucketing delay for the aggregation.

-- Trent

[Peter_Li]

I am using Python lately. But I have seen this with nodejs and rum-js.
I will be trying to use some of the agent tuning/queueing parameters to see if they improve the response speed.

I am using default kibana apm apps and drilldowns, with 15 min time window.
Don't know if that affects the aggregation time period.

Thanks

[trentm (Trent Mick)]

One config var for the Python agent that could make a bit of a difference is Configuration | APM Python Agent Reference [6.x] | Elastic
By default it sends APM events up to APM server every 10s.

The equivalent for the Node.js APM agent (Configuration options | APM Node.js Agent Reference [3.x] | Elastic) might not make a big difference. The Node.js agent streams up APM events as it captures them. It closes and restarts a streaming request to APM server every 30s by default.

At least for the Node.js agent setting the ELASTIC_APM_LOG_LEVEL=trace envvar will emit logs that show when events are sent to APM server -- though the output is a lot so may be difficult to trawl through.

[Peter_Li]

I think I found the problem..."user ignorance". I was using the kibana apm panel "throughput", that is not refreshed until enough time has passed. So I was waiting for 1-2 minutes before event shows up in the throughput aggregate metric (like you mentioned). Unfortunately, there doesn't seem to have a field in the apm dashboard showing the "most recent" transaction datetime. I think that would be very useful to see that with each transaction.

[trentm (Trent Mick)]

@Peter_Li

Unfortunately, there doesn't seem to have a field in the apm dashboard showing the "most recent" transaction datetime. I think that would be very useful to see that with each transaction.

Thanks for getting back! I've passed along the suggestion/idea to our UI folks.

[trentm (Trent Mick)]

Hi @Peter_Li,
I'm told that in the coming version 8.5 of the Elastic Stack that the transactions view will sort the matching transactions so the most recent is shown first. IIUC, that means that the first transaction shown in "Trace sample" should be the most recent. For example: