This issue was being caused because the indices were locked because of low disk space and was evident by running a GET .kibana/_settings and seeing a response similar to the following:
{
".kibana_2" : {
"settings" : {
"index" : {
"number_of_shards" : "1",
"auto_expand_replicas" : "0-1",
"blocks" : {
"read_only_allow_delete" : "true"
},
"provided_name" : ".kibana_1",
"creation_date" : "154192928788386",
"number_of_replicas" : "1",
"uuid" : "giheoigheo028r02810",
"version" : {
"created" : "6050499"
}
}
}
}
}
after freeing up disk space, you can do the following to unlock the .kibana index:
PUT /.kibana/_settings
{
"index.blocks.read_only_allow_delete": null
}
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/disk-allocator.html#disk-allocator has more details