Hi Guy,
We can skip this one. I have rebuilt the container. We'll see if I can break it again.
All was going well until mid-day yesterday. I had sucked most of the files into ES with a reasonably good pipeline.conf. Then bad things started to happen in the front-end. Initially, Kibana showed a locked out login page. With a message like 'Talk to your administrator.' After some stopping and restarting, I saw a bright red bar in the browser at localhost:5601, with message, "Kibana did not load properly. Check the server output for more information.". After more experiments, even that no longer appears.
I'm running the stack in the standard docker build on Mac OSX.
I've done some tests. Restarting Kibana never actually gets it running. Tried starting it both as a service and just running the bin/kibana executable.
netstat -natp | grep 5601 returns nothing.
curl result when trying to hit 5601:
root@4b04dc80a457:/# curl -XGET 'http://localhost:5601/'
curl: (7) Failed to connect to localhost port 5601: Connection refused
Elasticsearch is running:
root@4b04dc80a457:/# curl -XGET 'http://localhost:9200/'
{
"name" : "AsuTt_m",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "5wN3uC3MTXeKsQ3IvEYKYA",
"version" : {
"number" : "6.3.1",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "eb782d0",
"build_date" : "2018-06-29T21:59:26.107521Z",
"build_snapshot" : false,
"lucene_version" : "7.3.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
Here's the verbose error output when starting from bin/kibana
root@4b04dc80a457:/# /opt/kibana/bin/kibana > /var/log/kibana/stdout.txt
root@4b04dc80a457:/# cat /var/log/kibana/stdout.txt
{"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["plugin","debug"],"pid":419,"path":"/opt/kibana/node_modules/x-pack","message":"Found plugin at /opt/kibana/node_modules/x-pack"}
{"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["plugin","debug"],"pid":419,"path":"/opt/kibana/src/core_plugins/console","message":"Found plugin at /opt/kibana/src/core_plugins/console"}
<snipped - many plugins found>
{"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["plugin","debug"],"pid":419,"path":"/opt/kibana/src/core_plugins/tagcloud","message":"Found plugin at /opt/kibana/src/core_plugins/tagcloud"}
{"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["plugin","debug"],"pid":419,"path":"/opt/kibana/src/core_plugins/tile_map","message":"Found plugin at /opt/kibana/src/core_plugins/tile_map"}
{"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["plugin","debug"],"pid":419,"path":"/opt/kibana/src/core_plugins/timelion","message":"Found plugin at /opt/kibana/src/core_plugins/timelion"}
{"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["plugin","debug"],"pid":419,"path":"/opt/kibana/src/core_plugins/vega","message":"Found plugin at /opt/kibana/src/core_plugins/vega"}
{"type":"log","@timestamp":"2018-07-26T15:19:31Z","tags":["info","optimize"],"pid":419,"message":"Optimizing and caching bundles for ml, stateSessionStorageRedirect, status_page, timelion, graph, monitoring, dashboardViewer, apm and kibana. This may take a few minutes"}
{"type":"ops","@timestamp":"2018-07-26T15:19:34Z","tags":[],"pid":419,"os":{"load":[0.54296875,0.30810546875,0.26904296875],"mem":{"total":4139020288,"free":1063202816},"uptime":465537},"proc":{"uptime":7.073,"mem":{"rss":235671552,"heapTotal":191827968,"heapUsed":127149536,"external":236281},"delay":64.58339202404022},"load":{"requests":{},"concurrents":{"5601":0},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 121.3MB uptime: 0:00:07 load: [0.54 0.31 0.27] delay: 64.583"}
{"type":"ops","@timestamp":"2018-07-26T15:19:39Z","tags":[],"pid":419,"os":{"load":[0.57958984375,0.31982421875,0.2734375],"mem":{"total":4139020288,"free":1001254912},"uptime":465542},"proc":{"uptime":12.082,"mem":{"rss":296873984,"heapTotal":252522496,"heapUsed":206178264,"external":1599328},"delay":239.71886104345322},"load":{"requests":{},"concurrents":{"5601":0},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 196.6MB uptime: 0:00:12 load: [0.58 0.32 0.27] delay: 239.719"}
{"type":"ops","@timestamp":"2018-07-26T15:19:47Z","tags":[],"pid":419,"os":{"load":[0.64453125,0.3427734375,0.28125],"mem":{"total":4139020288,"free":952623104},"uptime":465550},"proc":{"uptime":19.877,"mem":{"rss":343646208,"heapTotal":295858176,"heapUsed":202410656,"external":4309377},"delay":48.28747296333313},"load":{"requests":{},"concurrents":{"5601":0},"responseTimes":{},"sockets":{"http":{"total":0},"https":{"total":0}}},"message":"memory: 193.0MB uptime: 0:00:20 load: [0.64 0.34 0.28] delay: 48.287"}
My google searches haven't yielded anything helpful. Maybe you can suggest some paths to explore.
Best,
Jim
I'm tempted to just pull our precious pipeline and beats configuration files out of the container, and start over with a fresh, new container!