Visualize: socket hangup

[babarsiddique]

Error: Bad Gateway
        at respond (http://xxx.xxx.xxx:5000/bundles/kibana.bundle.js?v=15543:13:2730)
        ........

Opening kibana's dashboard stopped the elasticsearch master and elasticsearch health goes from green to RED. It happens whenever the shards are more than approx 12000. Below this value dashboard works fine.

Anyone who can help me to resolve this issue?

[Brandon_Kobel]

Hey @babarsiddique, it sounds like you're querying too many shards and crashing Elasticsearch. 12,000 seems like a lot of shards. How large are your shards? The general guidance is to keep your shard size between a few GB and a few tens of GB per https://www.elastic.co/guide/en/elasticsearch/reference/7.4/scalability.html#it-depends.

[babarsiddique]

Hi @Brandon_Kobel, I have set index_number_shards and index_number_of_replicas to 1. Attached screenshot shows the details of my cluster. 1 elasticsearch master and 2 elasticsearch datanode....elasticsearch master is also act as elasticsearch datanode. Heap size for both nodes are 16g.

[Brandon_Kobel]

Hey @babarsiddique, those indices are super small, and each of them have 5 primary shards. I'd recommend reducing the primary shards to 1 and no longer user daily indices for this data, and instead using policies to manage index rollover: https://www.elastic.co/guide/en/elasticsearch/reference/7.4/using-policies-rollover.html

[babarsiddique]

Thanks @Brandon_Kobel, i have set the index_number_of_shards to 1 and index_number_of_replicas to 1 at the time of installation of elasticsearch but i think it is not creating any effect on indexes. For index rollover i have configured elasticsearch curator which deletes the old indices now i have around 8000 shards but dashboard again crashing the elasticsearch master.

Looking forward to hear from you
Thanks again!

[Brandon_Kobel]

Hey @babarsiddique, I'd recommend opening a new topic over in the Elastisearch section asking for advice on how to collapse individual indices into fewer. 8000 shards is still a ton, and executing a search which hits all of them is bound to cause issues.