Unassigned shards/indexes

[Jan_Engelbrecht]

Hi,
I'm quite new to Elastic, we're running 7.15.1 (Centos 7)on our production servers.
We have 3 nodes currently and the following happened when I added a 4th one.
After the 4th Node joined the cluster replication started and all looked good, but it got stuck on 97.5%, status stayed Yellow, eventually (after about 15min) I checked the logs and didn't see any errors.
I stopped node4 with the hope of just re-adding it would solve this replication stuck on 95%
After I stopped it, cluster health went to red.

...
curl http://10.0.3.16:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
`% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 7790 100 7790 0 0 511k 0 --:--:-- --:--:-- --:--:-- 543k
.ds-.logs-deprecation.Elasticsearch-default-2022.05.19-000001 0 p UNASSIGNED INDEX_REOPENED
.ds-.logs-deprecation.Elasticsearch-default-2022.05.19-000001 0 r UNASSIGNED REPLICA_ADDED
.ds-ilm-history-5-2022.05.19-000001 0 p UNASSIGNED INDEX_REOPENED
.ds-ilm-history-5-2022.05.19-000001 0 r UNASSIGNED INDEX_REOPENED
...

This is the shards that is not assigned, I tried closing and opening them and also tried removing and adding a replica (from what I read on the net that some people tried) and nothing works.
how can I go about to resolve this? The 2 indexes seems to be Elastic indexes.
How important are they?
Can I just delete them?
Is there a way to repair them?

Thanks very much
Jan

[dadoonet (David Pilato)]

IMO removing them should not cause any harm.

[Jan_Engelbrecht]

Hi,
Thanks for the reply, that is what I also thought, but when I try to remove it I get the following:

...
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "index [.ds-.logs-deprecation.Elasticsearch-default-2022.05.19-000001] is the write index for data stream [.logs-deprecation.Elasticsearch-default] and cannot be deleted"
}
],
"type" : "illegal_argument_exception",
"reason" : "index [.ds-.logs-deprecation.Elasticsearch-default-2022.05.19-000001] is the write index for data stream [.logs-deprecation.Elasticsearch-default] and cannot be deleted"
},
"status" : 400
}
...

[Jan_Engelbrecht]

I think i found the solution, for incase anyone else has this issue.
You need to roll_over too a new index.

...
curl -X POST "http://10.0.3.16:9200/.logs-deprecation.elasticsearch-default/_rollover/?pretty"
curl -X POST "http://10.0.3.16:9200/ilm-history-5/_rollover/?pretty"
...

After this you can delete the old ones.

Thanks