All my indexes "read_only_allow_delete" were set to true

[i333]

My system almost ran out of disk space so elasticsearch put itself into that offline mode or whatever it is. But when I cleared up the disk space and restarted elasticsearch, it started fine but I found that all my indexes "read_only_allow_delete" has been set to true and I can't write anything to the indexes!

Is this expected and normal behavior or did something go wrong?

Is it safe for me to just remove that setting? How can I do it?

Is there a way to do it for all indexes at once or I'll have to manually do it for all of them?

[Bernt_Rostad]

@i333

Is this expected and normal behavior or did something go wrong?

Yes, this is the normal behavior when a node is at risk of going full. According to the Disk-based Shard Allocation docs:

cluster.routing.allocation.disk.watermark.flood_stage

Controls the flood stage watermark. It defaults to 95%, meaning that Elasticsearch enforces a read-only index block ( index.blocks.read_only_allow_delete ) on every index that has one or more shards allocated on the node that has at least one disk exceeding the flood stage. This is a last resort to prevent nodes from running out of disk space. The index block must be released manually once there is enough disk space available to allow indexing operations to continue.

As you see from the last line, this index block must be manually released once you have full control of your cluster, with sufficient disc space on all nodes. You can remove the index block with the command listed in the documentation, something like this:

PUT /my_index/_settings
{
"index.blocks.read_only_allow_delete": null
}