Hi,
I reached disk usage of 90%.
I increased the disk space and now have 80% usage:
/dev/root 562G 445G 118G 80% /
I've tried to run the commands to remove the read only blocker:
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.disk.watermark.low": "90%",
"cluster.routing.allocation.disk.watermark.high": "95%",
"cluster.routing.allocation.disk.watermark.flood_stage": "97%"
}
}
PUT */_settings?expand_wildcards=all
{
"index.blocks.read_only_allow_delete": null
}
However ES still output the same error:
Feb 5 01:47:37 elastic1 logstash[1704888]: [2023-02-05T01:47:37,200][INFO ][logstash.outputs.elasticsearch][twitch_streams_in_progress][459ad5f58b8eabb83dd337c46ccfbb5d7908ed1b3d76624ce088d75da30b94b9] Retrying failed action {:status=>429, :action=>["update", {:_id=>"41290676350", :_index=>"stream_id", :routing=>nil, :_type=>"ts", :retry_on_conflict=>1}, {
.....
], :error=>{"type"=>"cluster_block_exception", "reason"=>"index [sid] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}}
Any idea what could be the blocker?
This is a single node running ES 7.17.9
I have just upgraded to 7.17 from 7.13.
I tried to add a document to an index using Kibana dev tools and it worked, but logstash keeps having this issue and does not push a single document