I am trying to repair an Elasticsearch / Logstash / Kibana (ELK) installation implemented by a former coworker. The system seemed to be running along fine until it ran out of space (several times now). I have successfully re-worked various .yml and/or config files to the point where I can now use curl from another system to take action. I have successfully deleted the oldest date of the logstash indices. So, I'm confident I can take action on the installation, if only I knew what I was doing!
curl 'http://hostname:9200/_cat/shards?pretty' will list 500+ lines of shards (indices?). All but four of them are "UNASSIGNED", and the other four are INITIALIZING, and never change.
curl 'http://hostname:9200/_cat/indices?pretty' lists every entry as "red open logstash-yyyy.mm.dd" and then two numbers after that.
I'm too new at this to know what's going on, or the correct action to take. As I said, I successfully deleted one of the logstash dates... good or bad, I did it.
The Kibana browser page shows status red, and gives some stats, and indicates the indexes are initializing, try again in 2.5 seconds.
What can I do to get this thing back to functioning, again?
Would running out of disk space (and I mean, zero bytes free) cause the symptoms I am seeing?
I asked all of the above on a different forum and got one reply, indicating I should come here, instead, and that a full disk might cause these problems, and that I might need to delete some more indices. I am a complete novice at this, and wouldn't know how to determine which indices might need to be deleted (which ones should I target to maintain as much data as I can?).
Any help at all would be appreciated. Here is a _cluster/health from my installation, so you can see how bad it really is.
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 4,
"unassigned_shards" : 526,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 4,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 30,
"active_shards_percent_as_number" : 0.0