ElasticSearch reindex suddenly cut docs by half and went red

Elastic Stack Elasticsearch
1

Hi, everyone.

I have a 25GB elasticsearch index and I have to add an analyzer to some fields, so I'm reindexing the entire index.

I wrote a ruby script that ran fine, but, this time, it behaved
oddly: when I was reaching the last docs (a bit over 10 million docs)
and, suddenly, my new index showed only half-ish the documents (5747957
out of 10902974-ish - and 13.34GB out of 25GB-ish) and went red. Funny
thing is the behaviour I watched on kopf plugin:

There is always one primary shard deallocated (as you can see, both
3's are unassigned, but they keep shifting to other shards every second
or so). On top of that, I've noticed this:

So, that disk space might be the issue too... and that one shard just won't initilize, it keeps that "1 shard initializing", but I'm not sure what's going on.

Could anyone shed some light on this issue?

Thanks in advance.

2

I suggest you close or delete the index. If you close it, it will leave the data there and you can re-open it at any time. With the index closed the cluster state will most likely go green. If thats the case, then you can delete the bad index, and try to re-index again.

3

did u try adding more disk space / disk (updating https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html#paths) so things settle down.

Btw - ES reports the document count available in active shards so if you have shard which is still not initialized/started, the documents in that shard will not show up.