How to collapse individual indices into fewer?

#1 
Error: Bad Gateway
        at respond (http://xxx.xxx.xxx:5000/bundles/kibana.bundle.js?v=15543:13:2730)
        ........

Opening kibana's dashboard stopped the elasticsearch master and elasticsearch health goes from green to RED. It happens whenever the shards are more than approx 8000.

I have set index_number_shards and index_number_of_replicas to 1. Attached screenshot shows the details of my cluster. 1 elasticsearch master and 2 elasticsearch datanode....elasticsearch master is also act as elasticsearch datanode. Heap size for both nodes are 16g.

Nodes

shards
shards.PNG922×111 7.15 KB

Any one who can help me to resolve it?
Thanks

#2

Some questions/hints:

It looks like it's still 5 shards per index? (Maybe I read the screenshot wrong) what does http(s)://yourelasticsearch:9200/specific-index-with-no-wildcards/_count return? (It should say X count, 1 shard, 1 successful)

Someone told me 1 shard is sufficient for storing ~25gb of data, so with your few megs per index, anything above 1 shard would be wasted. Probably because of this they changed the default from 5 to 1 shard per index with 7.x. The more shards, the more ram hungry the system becomes. I think(!) 800 shards is way too much for this size.

Do you delete old indices?

If possible try not to index into daily indices or aggregate the daily indices in monthly (or even bigger) ones after some time.

You can either do this aggregation by your own (check the _reindex-API) or check the Index Lifecycle Policy feature or maybe the Rollup feature. (Because I'm using Elasticsearch since 5.x I went the _reindex-way back then, the other features are new to me but look pretty cool and more advanced)