Kibana is running but extremely slow

Elastic Stack Kibana
1

Hi Fellows, i am suffering with this issue. I do not know how i could fix this. I would be glad if you could give a direction to go for it.

Kibana is working but extremely slow running. I have 10 nodes elasticsearch hot , cold, warm architecture cluster, haven't configured the ILM etc yet since this was a PoC environment. I have 3 master nodes including data roles and 7 dedicated data nodes (hot, warm, cold) but as i said haven't configured ILM yet for now so all data goes to 3 masters nodes. Discover tab is crying while loading itself rather than that other tabs like Dev Console \ Stack Management etc. reasonabily faster responding when click on it.

asd
asd1560×462 46 KB

customer is complaining about slowness, is there also jvm.options for kibana? what could be the reason? in every tab clicking on discover elastic logo is just loading and loading in the top left corner but taking a lot of minutes to load discover page or some other pages.

{"type":"log","@timestamp":"2021-07-28T13:25:15+03:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":100354,"message":"Error: Cluster client cannot be used after it has been closed.\n    at LegacyClusterClient.assertIsNotClosed (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:195:13)\n    at LegacyClusterClient.callAsInternalUser (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:115:12)\n    at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:22:18)\n    at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:209:43)\n    at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:195:20)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"}
{"type":"log","@timestamp":"2021-07-28T13:25:15+03:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":100354,"message":"Unable to bulk upload the stats payload to the local cluster"}
{"type":"log","@timestamp":"2021-07-28T13:25:25+03:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":100354,"message":"Error: Cluster client cannot be used after it has been closed.\n    at LegacyClusterClient.assertIsNotClosed (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:195:13)\n    at LegacyClusterClient.callAsInternalUser (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:115:12)\n    at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:22:18)\n    at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:209:43)\n    at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:195:20)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"}
{"type":"log","@timestamp":"2021-07-28T13:25:25+03:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":100354,"message":"Unable to bulk upload the stats payload to the local cluster"}
{"type":"log","@timestamp":"2021-07-28T13:25:35+03:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":100354,"message":"Error: Cluster client cannot be used after it has been closed.\n    at LegacyClusterClient.assertIsNotClosed (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:195:13)\n    at LegacyClusterClient.callAsInternalUser (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:115:12)\n    at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:22:18)\n    at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:209:43)\n    at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:195:20)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"}
{"type":"log","@timestamp":"2021-07-28T13:25:35+03:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":100354,"message":"Unable to bulk upload the stats payload to the local cluster"}
{"type":"response","@timestamp":"2021-07-28T13:24:35+03:00","tags":[],"pid":100354,"method":"post","statusCode":200,"req":{"url":"/api/telemetry/v2/clusters/_stats","method":"post","headers":{"host":"10.131.41.112:5601","connection":"keep-alive","content-length":"21","kbn-version":"7.11.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://10.131.41.112:5601","referer":"http://10.131.41.112:5601/app/discover","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9"},"remoteAddress":"10.6.219.233","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36","referer":"http://10.131.41.112:5601/app/discover"},"res":{"statusCode":200,"responseTime":64783,"contentLength":9},"message":"POST /api/telemetry/v2/clusters/_stats 200 64783ms - 9.0B"}
{"type":"log","@timestamp":"2021-07-28T13:25:45+03:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":100354,"message":"Error: Cluster client cannot be used after it has been closed.\n    at LegacyClusterClient.assertIsNotClosed (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:195:13)\n    at LegacyClusterClient.callAsInternalUser (/usr/share/kibana/src/core/server/elasticsearch/legacy/cluster_client.js:115:12)\n    at sendBulkPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/lib/send_bulk_payload.js:22:18)\n    at BulkUploader._onPayload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:209:43)\n    at BulkUploader._fetchAndUpload (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/bulk_uploader.js:195:20)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)"}
{"type":"log","@timestamp":"2021-07-28T13:25:45+03:00","tags":["warning","plugins","monitoring","monitoring","kibana-monitoring"],"pid":100354,"message":"Unable to bulk upload the stats payload to the local cluster"}
2

Heap memory space could be one of reasons for slowness.
Can you try and increase allocated heap space in jvm.options -
and check?

I also see that there are > 3000 shards for 3 master nodes. Any way to decrease them as each shard consumes lot of memory.One way is to reindex many indices into one as each index consumes 1 or more shards.

One other thing would be to see if any of your aggregation size in one of field in any of visualizations is greater than 3000.That may cause slowness.Try to lower if it is high.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.