Logs are coming through logstash just fine. There are no errors in the logstash log or the elasticsearch logs/indexing logs. I have restarted elasticsearch, logstash, and kibana, but the latest log shows 30 minutes old and I can't figure out why.
You said that the latest log is 30 min ago (51 by now) but what is the latest one in Logstash? Maybe it just doesn't have any new logs?
Oh they're coming through alright. As I'm tailing the logstash.stdout file, they are just flying through my screen.
Just trying out every small mistakes. The output conf of Logstash, does it point to the right host and port for elasticsearch?
Yes. Nothing in my configs have changed. It just all the sudden stopped showing logs in the kibana dashboard.
Here are the indices stats from elasticsearch. I'm not 100% sure what "yellow" means.
curl localhost:9200/_cat/indices yellow open logstash-2015.11.10 5 1 53238 0 14mb 14mb yellow open logstash-2015.11.14 5 1 463339 0 126.9mb 126.9mb yellow open logstash-2015.11.19 5 1 52235 0 15.6mb 15.6mb yellow open logstash-2015.11.15 5 1 462944 0 126.8mb 126.8mb yellow open logstash-2015.11.11 5 1 448247 0 111.8mb 111.8mb yellow open logstash-2015.11.16 5 1 467115 0 128.8mb 128.8mb yellow open logstash-2015.11.12 5 1 601605 0 162.4mb 162.4mb yellow open logstash-2015.11.17 5 1 562858 0 157.1mb 157.1mb yellow open logstash-2015.11.13 5 1 463421 0 128mb 128mb yellow open .kibana 1 1 2 1 11.5kb 11.5kb yellow open logstash-2015.11.18 5 1 199574 0 120.8mb 120.8mb
Indices are provided with a health status. This is something ELK provides on its own. Green, yellow, red and grey if I remember correctly. Also if I look at your indices I notice how there is a logstash-2015.11.19 among them.
For me at least it isn't the 19th yet but might be mistaken about that.
Regarding the logs not outputting, Maybe its an idea to check why the indices are yellow. Might not prove to be anything useful for your situation but you never know.
Ah it would seem that I was mistaken. There are actually countries now who are on the 19th.
The date on the server was incorrect. That's why theres a logstash-2015.11.19. I've since corrected the date.
I just rebooted the server and now things are coming through kibana again. I have no idea what happened, but it seems to be fixed.
Surprisingly that was my next suggestion. Seems like the old "Did you turn it off and on again" never fails.
Weird how it didn't work at first.