Kibana APM App pages are TOO SLOW

Elastic Observability APM
1

Kibana APM App pages are too slow with only 1000 tps transaction data.
Elasticsearch is installed on 10 'n1-highcpu-16' nodes on Google Compute Engine.
I captured the screen to see the loading latency.

Kibana version:
7.5.0~7.6.0

Elasticsearch version:
7.5.0~7.6.0

APM Server version:
7.5.0~7.6.0

APM Agent language and version:
java

Browser version:
Chrome (80.0.3987.116 (64bit))

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
ECK, download page.

Fresh install or upgraded from other version?
7.5.0 (Fresh install)
7.6.0 (upgraded from 7.5.0)

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

  • Index Pattern :
    • primary shard : 5
    • refresh interval : 10s
  • sample rate : 0.01

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:

  1. Keep call the service with 1000 tps load.
  2. wait for 24 hours for the apm data load to the elasticsearch nodes.
  3. Open Kibana apm pages. It take more than 20seconds for loading most of APM pages.
  4. The issue can be reproduced with 100% possibility on any environment.
    • eck / 10 nodes(16core, 16G ram)/5 primary shard, 1replica
    • normal / 10 nodes(16core, 16G ram)/5 primary shard, 1replica

Errors in browser console (if relevant):
it took more than 20seconds to load every APM page.

Provide logs and/or server output (if relevant):

2

Hi there,

Thanks for the good description and screen capture. It clearly looks very slow.
To get more insights into the slow requests you should enable some additional debugging. By appending &_debug=true to the url of the slow request, the exact Elasticsearch query that is executed will be logged in the Kibana process.

Example: /api/apm/services?start=2020-02-23T11%3A50%3A01.650Z&end=2020-02-24T11%3A50%3A01.650Z&uiFilters=%7B%7D&_debug=true

Please copy the ES query into Kibana Dev Tools and verify whether it's still performing poorly, then post the query here.

Thank you.

1 Like
3

Hello,

I tried what you said to find the cause of the slow loading time.
However, every queries in the Kibana APM App seems to very slow.

I captured all queries and made a google sheet.
Elastic APM Slow Queries Google Sheet

All data is on 'Time' sheet, and on Page1, Page2, Page3 sheets I put a captured Kibana screen with Chrome dev tool result.

One more thing I want to ask is,

I found THIS APM Issue on APM project, and this issue is quite related to the APM latency problem. Is it right?

4

Thanks for capturing this information. Much appreciated! I'll need a bit of time to look into it.

I found THIS APM Issue on APM project, and this issue is quite related to the APM latency problem. Is it right?

Yes, absolutely. That's one approaches we are looking into to making the IU faster.

5

Okay :slight_smile:

Take your time, and I hope you to find the cause of the problem.

6

Hello @sqren

Any updates on this issue?

7

Adding related issues

8

Hi,
This issue might be helpful to you - APM transaction data not compressed properly.
APM is planning to release a fix in 7.8 - https://github.com/elastic/apm-server/issues/3485

1 Like
9

Thanks you

10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.