Issue with elastic cluster after upgrade to 6.5.0

[planetvortex (Tony S.)]

My master node is reporting the following error "role retrieval failed from [native roles store]"

[2018-11-21T18:27:50,739][WARN ][o.e.x.s.a.s.CompositeRolesStore] [elk-master-2.mydomain.net] role retrieval failed from [native roles store]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

I can no longer view roles, i can list them, but not view the individual contents of any of the roles, nor can I create users.
I've tried my own account, and have tried the elastic user. The upgrade may not be the issue.
I'm not sure what to provide to you.

[planetvortex (Tony S.)]

OK, so this may be the wrong way to handle things.

PUT .*/_settings
{
"index": {
"blocks": {
"read_only_allow_delete": "false"
}
}
}

PUT */_settings
{
"index": {
"blocks": {
"read_only_allow_delete": "false"
}
}
}

[Christian_Dahlqvist]

That error message sometimes indicates that you are running out of disk space and have exceeded the flood stage watermark (defaults to 95% full). If that is the case, you will need to free up space as indices otherwise will go back to read-only mode.

[planetvortex (Tony S.)]

I WAS having an issue with the space, which I resolved, and the error never cleared itself. Of course by the time I cleared things up, the monitoring wouldn't run anymore.

[planetvortex (Tony S.)]

How long should it take to become NOT read-only after a space issue has been cleared?

[Christian_Dahlqvist]

I believe you need to clear it manually like you did before.

[planetvortex (Tony S.)]

Thank you, the cluster has cleared. I learned a new trick with this too and led me to the discover of this:
GET _cluster/allocation/explain

I had a lot of nodes with the wrong version of elasticsearch as well. They must've changed version in the yum repo the next day after I rebuilt the other nodes.