Kibana Crash while loading index list

[dslavescu (Diana Slavescu)]

Hello,

I have an ELK Cluster of 3M+12D nodes , with 64GB and 8 cores for the data nodes and a total of 53 TB of data, 3243 indices, 7602 shards. On top, i have a Kibana instance of 8GB RAM + 4 CPU. Both Kibana and ELK are at 7.17.2 version.
From Kibana when i try to go to Stack Management -> Index Management -> Indices while i wait for the index list to be loaded, kibana service crashes with the following message in service logs:

Feb 06 09:39:41 localhost kibana[799]: /opt/application/Kibana/kibana-7.12.1-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:195
Feb 06 09:39:41 localhost1 kibana[799]:       response += d;
Feb 06 09:39:41 localhost kibana[799]:                   ^
Feb 06 09:39:41 localhost kibana[799]: RangeError: Invalid string length
Feb 06 09:39:41 localhost kibana[799]:     at IncomingMessage.<anonymous> (/opt/application/Kibana/kibana-7.12.1-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:195:19)
Feb 06 09:39:41 localhost1 kibana[799]:     at IncomingMessage.emit (events.js:315:20)
Feb 06 09:39:41 localhost kibana[799]:     at addChunk (internal/streams/readable.js:309:12)
Feb 06 09:39:41 localhost kibana[799]:     at readableAddChunk (internal/streams/readable.js:280:11)
Feb 06 09:39:41 localhost kibana[799]:     at IncomingMessage.Readable.push (internal/streams/readable.js:223:10)
Feb 06 09:39:41 localhost kibana[799]:     at HTTPParser.parserOnBody (_http_common.js:139:24)
Feb 06 09:39:41 localhost kibana[799]:     at HTTPParser.execute (<anonymous>)
Feb 06 09:39:41 localhost kibana[799]:     at Socket.socketOnData (_http_client.js:509:22)
Feb 06 09:39:41 localhost kibana[799]:     at Socket.emit (events.js:315:20)
Feb 06 09:39:41 localhost kibana[799]:     at addChunk (internal/streams/readable.js:309:12)
Feb 06 09:39:41 localhost kibana[799]:     at readableAddChunk (internal/streams/readable.js:284:9)
Feb 06 09:39:41 localhost kibana[799]:     at Socket.Readable.push (internal/streams/readable.js:223:10)
Feb 06 09:39:41 localhost kibana[799]:     at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
Feb 06 09:39:41 localhost systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Feb 06 09:39:41 localhost systemd[1]: kibana.service: Failed with result 'exit-code'.

Is there a limitation in how many indices can be listed into the Kibana UI?
Did anyone else encountered this issue and can help?

Thanks!

[warkolm (Mark Walkom)]

FWIW 7.12 is EOL and no longer supported, you should be looking to upgrade as a matter of urgency. Also it looks like your average shard size is about 6-7GB, which is pretty small and you are likely wasting some resources with that.

To your question - I haven't seen this before, but the first and best option is to upgrade to 7.17 to see if that helps.