I am not sure what went wrong, kibana visualization works without issue, but here is the status of the cluster:
and
No errors in production.log on any of the data nodes. Restarted elasticsearch on data nodes without success.
There are 4 unassigned shards. You can probably find them with the _cat/shards api. I'm going to take a wild guess and say that at some point you lost the last primary copy of one of those shards. Elasticsearch can restore it because the data is just gone. This can happen if you create the index without any replicas and you lose a node. If you don't need the index you can just curl -XDELETE localhost:9200/index_name and it'll be gone for good. If you want to assign that shard you can use the cluster reroute API with allow_primary set to true. That'll cause an empty shard to be created but it'll get the cluster green again. That is useful if you lost only a single shard in a multishard index and you want create it empty and just reindex all the data that needs to be in there.
1 Like