Unable to create index pattern from Kibana

[jmckeown (James McKeown)]

I am experiencing a problem similar in nature to the one here:

Kibana creating new index forever

~Hello, I have an issue creating new Index in Kibana 6.4.0; I successfully create logs with indexes on logstash - Kibana see's the inside of those logs in Dev Tolls - but for some reason when I go to Management->Index Patterns -> Create new Index - and I pick the indexes I'm looking for the circle for 'creating' is spinning forever- I've left it for 7 hours without result - I even stopped logstash when logs had only few MB - still it was spinning constantly. …

I have written a few (five) documents to an index, but when I attempt to make an index pattern through Kibana, the wheel just spins.

As per @jbudz's request, here are my kibana settings.

{".kibana":{"settings":{"index":{"number_of_shards":"1","auto_expand_replicas":"0-1","blocks":{"read_only_allow_delete":"true"},"provided_name":".kibana","creation_date":"1541011008451","number_of_replicas":"0","uuid":"45i6FyJUQTad-fes0s72vg","version":{"created":"6040299"}}}}}

[rashmi (kulkarni)]

@jmckeown

"read_only_allow_delete":"true"

I think you are hitting high threshold and hence its becoming read only indices. Can you free up some space to see if this gets resolved ?

All indexes get locked when the disk threshold is reached

To unlock all indexes manually:

curl -XPUT -H "Content-Type: application/json" https://[YOUR_ELASTICSEARCH_ENDPOINT]:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Also try deleting .kibana index to get it back up and going.

Hope it helps
Cheers
Rashmi

[jmckeown (James McKeown)]

I think that was the problem. high disk watermark [90%] exceeded on [fcqHgNaCRUWBvF5fvgxl_A][fcqHgNa][/usr/local/var/lib/elasticsearch/nodes/0] free: 13.7gb[5.8%], shards will be relocated away from this node

[rashmi (kulkarni)]

Am glad that the root cause is know.. Cheers