Error: Bad Gateway
at respond (http://xxx.xxx.xxx:5000/bundles/kibana.bundle.js?v=15543:13:2730)
........
Opening kibana's dashboard stopped the elasticsearch master and elasticsearch health goes from green to RED. It happens whenever the shards are more than approx 8000.
I have set index_number_shards and index_number_of_replicas to 1. Attached screenshot shows the details of my cluster. 1 elasticsearch master and 2 elasticsearch datanode....elasticsearch master is also act as elasticsearch datanode. Heap size for both nodes are 16g.
It looks like it's still 5 shards per index? (Maybe I read the screenshot wrong) what does http(s)://yourelasticsearch:9200/specific-index-with-no-wildcards/_count return? (It should say X count, 1 shard, 1 successful)
Someone told me 1 shard is sufficient for storing ~25gb of data, so with your few megs per index, anything above 1 shard would be wasted. Probably because of this they changed the default from 5 to 1 shard per index with 7.x. The more shards, the more ram hungry the system becomes. I think(!) 800 shards is way too much for this size.
Do you delete old indices?
If possible try not to index into daily indices or aggregate the daily indices in monthly (or even bigger) ones after some time.
You can either do this aggregation by your own (check the _reindex-API) or check the Index Lifecycle Policy feature or maybe the Rollup feature. (Because I'm using Elasticsearch since 5.x I went the _reindex-way back then, the other features are new to me but look pretty cool and more advanced)
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.