Getting 404 shard_not_found_exception frequently

[ravib]

I have an index with 2 replicas and 1 shard on a 3 node elastic search server (7.6.1). On bulk requests (index) I keep getting shard_not_found_exception, any thoughts?

[Charele]

How about health of you cluster?

I think, you can decrease you replicas of this index to 0,and test the bulk
(if it's okay, restore the replicas to 2)

[warkolm (Mark Walkom)]

It'd be helpful if you shared the log that contains this error.

[ravib]

This happens once every few months, so I usually end up creating a new index and move all the data into it, which fixes it.

@Charele cluster health is good (green), I tried bulk request and also _reindex API with zero replicas, it worked fine, it fails randomly

@warkolm this is the response from the bulk request
{"_index":"xxx","_type":"_doc","_id":"xxx","status":404,"error":{"type":"shard_not_found_exception","reason":"no such shard","index_uuid":"xxx","shard":"2","index":"xxx"}}

I have not seen anything specific for this in logs, but I had another frequent issue (which I fixed), do you this could have caused it?

Caused by: java.lang.IllegalArgumentException: Field [xxx] of type [dense_ve
ctor] of doc [xxx] has number of dimensions [0] l
ess than defined in the mapping [40]


[2020-08-13T16:42:26,418][DEBUG][o.e.a.b.TransportShardBulkAction] [elastic7-03] [index_name][0] failed to execute bulk item (index) index {[index_name][_doc][doc_id], source[n/a, actual length: [2.5kb], max length: 2kb]}