No I have tons of gb left. I went into /var/log/elasticsearch and wrote "ls" to get a list of the items. I even wrote ls -li to get a more structured list but then it said "total 0"
How are you starting ES? With what arguments (ps aux | grep elasticsearch will do)?
I'm actually running elasticsearch, logstash and kibana in a docker container so they become their own processes. I have a premade configuration file to start them all. I am starting elasticsearch with "./es"
Sorry, I'm out of patience here. The ES command line probably contains the log file path. Either way, locate the log file (elk.log in your case). Good luck.
Thank you. I just solved it although I don't think it is the way you are supposed to solve it. I went into the elasticsearch configuration file and changed the cluster.name to something else. I figured I'd give it a try because the cluster was red.
Sorry if I disturbed you in some way. I panicked because I'm pretty new to elastic and apparently it was much bigger than I thought.
I had same problem "retrying failed action with response code: 503", :level=>:warn}" and was able to resolved it. Hope below info will be help full to others.
My Env is elasticsearch-1.7.1 kibana-4.1.1-linux-x64 logstash-1.5.4
In ES web interface check the health
http://:9200/_plugin/marvel/sense/index.html
GET _cluster/health?level=indices&pretty
Stop ES and Kibana.
You might notice few indices "status":"red" - Example .marvel-2015.09.22
There will be *.recovery files inside indices folder.
Remove them and other indices after that date.
- Example .marvel-2015.09.23
.marvel-2015.09.24
And then try to start Elastic search, Kibana
and then load data using logstash
./logstash -f ../../../logstash-config/CRev_db2_config_v1.0.cfg --configtest
./logstash -f ../../../logstash-config/CRev_db2_config_v1.0.cfg