Hello. We upgraded our Elastic stack (Elasticsearch nodes, logstash, beats, kibana) from 7.13 to 7.15.0.
Everything is working well except that in Kibana, Stack Monitoring (Cluster Overview) we are getting some weird timeouts when trying to view the Elasticsearch "Overview" of a time-period longer than 30 minutes (the page that shows the overview stats of search rate and latency, indexing rate and latency). The page just keeps loading but no content is shown (header, banner and menu of Kibana GUI is only shown with "Loading..." text).
Last 15 minutes or Last 30 minutes i okay, but a longer period than that and it is not working anymore.
Everything else under Cluster Overview is working- I can see Elasticsearch nodes or any other overview of any time period under Cluster overview.
Usually Kibana GUI shows a message after some time of waiting:
Monitoring Request Error
An HTTP request has failed to connect. Please check if the Kibana server is running and that your browser has a working connection, or contact your system administrator.
Kibana logs constantly give an error during waiting for page to load:
{"type":"log","@timestamp":"2021-10-08T09:06:26+03:00","tags":["error","plugins","monitoring","monitoring"],"pid":598389,"message":"TimeoutError: Request timed out\n at ClientRequest.onTimeout (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Connection.js:110:16)\n at ClientRequest.emit (events.js:400:28)\n at TLSSocket.emitRequestTimeout (_http_client.js:790:9)\n at Object.onceWrapper (events.js:519:28)\n at TLSSocket.emit (events.js:412:35)\n at TLSSocket.Socket._onTimeout (net.js:484:8)\n at listOnTimeout (internal/timers.js:557:17)\n at processTimers (internal/timers.js:500:7) {\n meta: {\n body: null,\n statusCode: null,\n headers: null,\n meta: {\n context: null,\n request: [Object],\n name: 'elasticsearch-js',\n connection: [Object],\n attempts: 3,\n aborted: false\n }\n }\n}"}
Any tips/ideas what might be wrong? Should I investigate relaxing some timeout timers? We did not have this problem on 7.13. Thanks!