I am using Elasticsearch 5.0.1 and suddenly Seeing the Elasticsearch plugin is Red and Service unavailable.
I have heartbeat, filebeat and metricbeat in the Elasticsearch. I could see data for metribeat but no data in the discover for filebeat and heartbeat.
How to debug and fix this error.
please suggest.
Kibana log:
1 {"type":"log","@timestamp":"2018-04-18T10:03:07Z","tags":["status","plugin:kibana@5.0.1","info"],"pid":16176,"state":"green","message":"Status changed from unin itialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
2 {"type":"log","@timestamp":"2018-04-18T10:03:07Z","tags":["status","plugin:elasticsearch@5.0.1","info"],"pid":16176,"state":"yellow","message":"Status changed f rom uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
3 {"type":"log","@timestamp":"2018-04-18T10:03:08Z","tags":["status","plugin:console@5.0.1","info"],"pid":16176,"state":"green","message":"Status changed from uni nitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
4 {"type":"log","@timestamp":"2018-04-18T10:03:08Z","tags":["status","plugin:elasticsearch@5.0.1","info"],"pid":16176,"state":"green","message":"Status changed fr om yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
5 {"type":"log","@timestamp":"2018-04-18T10:03:09Z","tags":["status","plugin:timelion@5.0.1","info"],"pid":16176,"state":"green","message":"Status changed from un initialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
6 {"type":"log","@timestamp":"2018-04-18T10:03:09Z","tags":["listening","info"],"pid":16176,"message":"Server running at http://0.0.0.0:5601"}
7 {"type":"log","@timestamp":"2018-04-18T10:03:09Z","tags":["status","ui settings","info"],"pid":16176,"state":"green","message":"Status changed from uninitialize d to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
8 {"type":"response","@timestamp":"2018-04-18T10:03:13Z","tags":,"pid":16176,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers" :{"host":"kibana-server.com","accept":"text/html, application/xhtml+xml, /","referer":"https://kibana-server.com/kibana","acce pt-language":"en-US","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko","accept-encoding":"gzip, deflate","dnt":"1","x-forwarde d-proto":"https","x-forwarded-port":"443","x-forwarded-for":"104.254.254.3","x-forwarded-host":"kibana-server.com","x-forwarded-server":"kibana-server","connection":"Keep-Alive"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://kibana-server.com/ kibana"},"res":{"statusCode":200,"responseTime":577,"contentLength":9},"message":"GET /app/kibana 200 577ms - 9.0B"}
After restarting Kibana and heartbeat, hearbeat data is available in Kibana. I see some error in filebeat. Do you think this could have caused filebeat not to work.
2018-04-18T10:49:17+02:00 ERR Connecting error publishing events (retrying): dial tcp 192.192.1.1:5044: getsockopt: connection refused
2018-04-18T10:49:33+02:00 ERR Connecting error publishing events (retrying): dial tcp 192.192.1.1:5044: getsockopt: connection refused
2018-04-18T11:05:10+02:00 ERR Failed to publish events caused by: read tcp 167.29.421.18:3398->192.192.1.1:5044: i/o timeout
2018-04-18T11:05:41+02:00 ERR Failed to publish events caused by: read tcp 167.29.421.18:3598->192.192.1.1:5044: i/o timeout