Looks like I have been hit with a rather common issue. Though in my case I am not using any third party plugins.
Details:
Machine: Red Hat Enterprise Linux Server release 7.4
RAM: 8GB
It also hosts Elasticsearch (It is not production setup)
Action: Upgraded Kibana from 6.3.2-1 to 6.5.1-1. Upgrade went smooth
Result: Kibana never came up. On checking the logs this was the message at which it was stuck. This message kept getting repeated after some lengthy interval.
{"type":"log","@timestamp":"2018-11-22T11:39:50Z","tags":["plugin","warning"],"pid":18789,"path":"/usr/share/kibana/src/core_plugins/spy_modes","message":"Skipping non-plugin directory at /usr/share/kibana/src/core_plugins/spy_modes"}
{"type":"log","@timestamp":"2018-11-22T11:39:51Z","tags":["info","optimize"],"pid":18789,"message":"Optimizing and caching bundles for ml, stateSessionStorageRedirect, status_page, timelion, graph, monitoring, space_selector, dashboardViewer, apm, canvas, infra and kibana. This may take a few minutes"}
To get some more details I tried the verbose trick.
/usr/share/kibana/bin/kibana --verbose -c /etc/kibana/kibana.yml
This is what I got.
{"type":"log","@timestamp":"2018-11-22T03:25:10Z","tags":["debug","http","server"],"pid":20336,"message":"http server running at http://helloworld:5998"}
{"type":"log","@timestamp":"2018-11-22T03:25:10Z","tags":["debug","legacy","service"],"pid":20336,"message":"starting legacy service"}
{"type":"log","@timestamp":"2018-11-22T03:25:11Z","tags":["debug","config"],"pid":20336,"message":"Marking config path as handled: server"}
{"type":"log","@timestamp":"2018-11-22T03:25:11Z","tags":["debug","legacy","proxy"],"pid":20336,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2018-11-22T03:25:11Z","tags":["debug","legacy","proxy"],"pid":20336,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2018-11-22T03:25:22Z","tags":["plugin","debug"],"pid":20336,"path":"/usr/share/kibana/node_modules/x-pack","message":"Found plugin at /usr/share/kibana/node_modules/x-pack"}
{"type":"log","@timestamp":"2018-11-22T03:25:11Z","tags":["debug","legacy","service"],"pid":20336,"message":"Kibana server is not ready yet POST:/elasticsearch/_msearch."}
{"type":"log","@timestamp":"2018-11-22T03:25:22Z","tags":["debug","legacy","proxy"],"pid":20336,"message":"\"getConnections\" has been called."}
.
.
.
.
{"type":"log","@timestamp":"2018-11-22T03:26:54Z","tags":["debug","legacy","proxy"],"pid":20336,"message":"\"getConnections\" has been called."}
{"type":"ops","@timestamp":"2018-11-22T03:26:54Z","tags":[],"pid":20336,"os":{"load":[1.78955078125,1.2578125,1.49365234375],"mem":{"total":8203014144,"free":736821248},"uptime":2904375},"proc":{"uptime":105.087,"mem":{"rss":789721088,"heapTotal":731795456,"heapUsed":469409224,"external":28132358},"delay":106.64260482788086},"load":{"requests":{},"concurrents":{"5601":2},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 447.7MB uptime: 0:01:45 load: [1.79 1.26 1.49] delay: 106.643"}
{"type":"log","@timestamp":"2018-11-22T03:26:59Z","tags":["debug","legacy","proxy"],"pid":20336,"message":"\"getConnections\" has been called."}
{"type":"ops","@timestamp":"2018-11-22T03:26:59Z","tags":[],"pid":20336,"os":{"load":[1.7265625,1.25341796875,1.4912109375],"mem":{"total":8203014144,"free":719556608},"uptime":2904380},"proc":{"uptime":110.098,"mem":{"rss":806907904,"heapTotal":751693824,"heapUsed":420399104,"external":24737116},"delay":46.86207103729248},"load":{"requests":{},"concurrents":{"5601":2},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 400.9MB uptime: 0:01:50 load: [1.73 1.25 1.49] delay: 46.862"}
{"type":"log","@timestamp":"2018-11-22T03:27:04Z","tags":["debug","legacy","proxy"],"pid":20336,"message":"\"getConnections\" has been called."}
{"type":"ops","@timestamp":"2018-11-22T03:27:04Z","tags":[],"pid":20336,"os":{"load":[1.66845703125,1.2490234375,1.48876953125],"mem":{"total":8203014144,"free":702382080},"uptime":2904385},"proc":{"uptime":115.111,"mem":{"rss":823590912,"heapTotal":767447040,"heapUsed":585194408,"external":25700873},"delay":16.464394092559814},"load":{"requests":{},"concurrents":{"5601":2},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 558.1MB uptime: 0:01:55 load: [1.67 1.25 1.49] delay: 16.464"}
{"type":"log","@timestamp":"2018-11-22T03:27:11Z","tags":["debug","legacy","proxy"],"pid":20336,"message":"\"getConnections\" has been called."}
{"type":"ops","@timestamp":"2018-11-22T03:27:12Z","tags":[],"pid":20336,"os":{"load":[1.5654296875,1.2412109375,1.48388671875],"mem":{"total":8203014144,"free":629055488},"uptime":2904391},"proc":{"uptime":121.483,"mem":{"rss":896516096,"heapTotal":839262208,"heapUsed":505503048,"external":19850987},"delay":804.1342277526855},"load":{"requests":{},"concurrents":{"5601":2},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 482.1MB uptime: 0:02:01 load: [1.57 1.24 1.48] delay: 804.134"}
{"type":"log","@timestamp":"2018-11-22T03:27:14Z","tags":["debug","legacy","proxy"],"pid":20336,"message":"Event is being forwarded: connection"}
{"type":"log","@timestamp":"2018-11-22T03:27:14Z","tags":["debug","legacy","service"],"pid":20336,"message":"Kibana server is not ready yet POST:/elasticsearch/_msearch."}
{"type":"log","@timestamp":"2018-11-22T03:27:16Z","tags":["debug","legacy","proxy"],"pid":20336,"message":"\"getConnections\" has been called."}
{"type":"ops","@timestamp":"2018-11-22T03:27:16Z","tags":[],"pid":20336,"os":{"load":[1.52001953125,1.2373046875,1.4814453125],"mem":{"total":8203014144,"free":614453248},"uptime":2904396},"proc":{"uptime":126.511,"mem":{"rss":910381056,"heapTotal":854966272,"heapUsed":541285688,"external":18662628},"delay":5.138576030731201},"load":{"requests":{},"concurrents":{"5601":2},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 516.2MB uptime: 0:02:07 load: [1.52 1.24 1.48] delay: 5.139"}
I am kind of confused as how to proceed. There should be some error message hidden somewhere in the logs which is some how escaping me.
EDIT: I did use to have memory issue and there was this time when indices got marked as read only. When it was not working then as a last resort I tried this command on also but of no avail.
curl -X PUT "helloworld:5998/*/_settings" -H 'Content-Type: application/json' -d' {"index.blocks.read_only_allow_delete": null}'
