Kibana not starting up

Elastic Stack Kibana
1

I am trying to run a stack on my ubuntu docker system, and I tried many different containers but I simply cannot get it to work.
The last one i tried,,, hence this discussion is :elastic/stack-docker
When I run that.. I cannot even het kibana to showup on the port 5601
It was suggested to increase the memory, but how do I do that on an ubuntu machine ? I thought docker had to complete memory

Any ideas why kibana does not show its dashboard ?

{"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["status","plugin:kibana@6.1.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":" uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["status","plugin:elasticsearch@6.1.0","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch:9200/ => connect ECONNREFUSED 172.21 .0.2:9200"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["status","plugin:xpack_main@6.1.0","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for El asticsearch","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["status","plugin:graph@6.1.0","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elastic search","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["status","plugin:xpack_main@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elast icsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["status","plugin:graph@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsea rch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["status","plugin:elasticsearch@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to El asticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["error","elasticsearch","data"],"pid":1,"message":"Request error, retrying\nGET http://elasticsearch:9200/_xpack => connect ECONNREFUSED 17 2.21.0.2:9200"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["status","plugin:monitoring@6.1.0","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevStat e":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["warning","elasticsearch","data"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"} {"type":"log","@timestamp":"2017-12-26T09:28:05Z","tags":["warning","elasticsearch","data"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2017-12-26T09:28:13Z","tags":["reporting","warning"],"pid":1,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent pending reports from failin g on restart, please set xpack.reporting.encryptionKey in kibana.yml"} {"type":"log","@timestamp":"2017-12-26T09:28:13Z","tags":["status","plugin:reporting@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from uninitialized to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"uninitialized","prevMsg":"uninitialized"} {"type":"log","@timestamp":"2017-12-26T09:28:13Z","tags":["license","warning","xpack"],"pid":1,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: No Living connections"} {"type":"log","@timestamp":"2017-12-26T09:28:13Z","tags":["status","plugin:xpack_main@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from red to red - No Living connections","prevS tate":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch:9200."} {"type":"log","@timestamp":"2017-12-26T09:28:13Z","tags":["status","plugin:graph@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from red to red - No Living connections","prevState" :"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch:9200."} {"type":"log","@timestamp":"2017-12-26T09:28:13Z","tags":["status","plugin:reporting@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from red to red - No Living connections","prevSt ate":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch:9200."} {"type":"log","@timestamp":"2017-12-26T09:28:13Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"} {"type":"log","@timestamp":"2017-12-26T09:28:13Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"} {"type":"log","@timestamp":"2017-12-26T09:28:19Z","tags":["status","plugin:xpack_main@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from red to red - Request Timeout after 3000ms" ,"prevState":"red","prevMsg":"No Living connections"} {"type":"log","@timestamp":"2017-12-26T09:28:19Z","tags":["status","plugin:graph@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from red to red - Request Timeout after 3000ms","pre vState":"red","prevMsg":"No Living connections"} {"type":"log","@timestamp":"2017-12-26T09:28:19Z","tags":["status","plugin:reporting@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from red to red - Request Timeout after 3000ms", "prevState":"red","prevMsg":"No Living connections"} {"type":"log","@timestamp":"2017-12-26T09:28:19Z","tags":["status","plugin:elasticsearch@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from red to red - Request Timeout after 3000 ms","prevState":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch:9200."} {"type":"log","@timestamp":"2017-12-26T09:28:21Z","tags":

2

Kibana's trying to connect to elasticsearch but it can't find it. Do you have elasticsearch running in a container too and can you share your docker network config?

Can you share your install steps, I can try and reproduce it. Memory should be fine, the docker images come pre-bundled so you'll only need ~200mb to run the server.

3

I am using exactly this set: https://github.com/elastic/stack-docker

Your assistance is greatly appreciated.... as trying to setup the elk stack is driving me nuts now....

4

This is also in the kibana logs :
{"type":"log","@timestamp":"2017-12-28T07:32:59Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch:9200/ => connect ECONNREFUSED 172.21.0.2:9200"}

5

When I do this from outside my docker containers

curl -u elastic:changeme http://elasticsearch:9200
I get :
curl: (6) Could not resolve host: elasticsearch

when I do
curl -u elastic:changeme http://172.21.0.2:9200
then I get the version information back

maybe this helps....

6

I was able to reproduce this, looks like a bug. I filed https://github.com/elastic/stack-docker/issues/21.

1 Like
7

This is how user-defined networks work in Docker. You can actually take a look at the networking setup created by the docker-compose file:

  • Run docker network ls
  • Copy the ID of the network stackdocker_stack
  • Run docker network inspect <ID>

From your host you can access Elasticsearch and Kibana on 127.0.0.1 or localhost (with the respective port). Inside the Docker network, you can access them with the internal DNS names, which comes down to elasticsearch and kibana.

Now coming back to your original issue:

  1. You only need to increase the memory allocated to the Docker daemon on Mac and Windows. Since you're on Linux, you don't have to do anything, but you need to have enough memory available. Do you have 4GB of memory just for the containers? Lack of RAM is the most common reason we see for Kibana not coming up correctly.
  2. The logs you're showing are for 25s or so. I doubt this is enough for the entire stack to come up, configure itself, and reach a stable status. Please show enough logs until the stack has finished starting up (I'd estimate at least 2min) if Elasticsearch is running fine; otherwise we'd have to start with Elasticsearch.

PS: Docker looks very simple to get started with, however it adds a lot of complexity that is initially hidden. If you are not too familiar with Docker, I would try to get started without it, since it will double the learning curve.

1 Like
8

Great that it worked for you.... can you assist me in getting to to work on my ubuntu ?

9

See my post directly above. The 2 points about memory and logs are for your problem.

10

I dove a little deeper... even though I have 3.8G of memory... there is not alot of free mem left... let me test releasing some mem

11

After killing all my existing containers I now have 3G free mem...
Kibana is still not coming up
First 5 min Error logs are below
First 5 min complete log... how can I send that to you ?

{"type":"log","@timestamp":"2017-12-31T12:09:11Z","tags":["error","elasticsearch","admin"],"pid":1,"message":"Request error, retrying\nHEAD http://elasticsearch:9200/ => connect ECONNREFUSED 172.21.0.2:9200"}
{"type":"log","@timestamp":"2017-12-31T12:09:11Z","tags":["status","plugin:xpack_main@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2017-12-31T12:09:11Z","tags":["status","plugin:graph@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2017-12-31T12:09:11Z","tags":["status","plugin:elasticsearch@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Unable to connect to Elasticsearch at http://elasticsearch:9200.","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2017-12-31T12:09:11Z","tags":["error","elasticsearch","data"],"pid":1,"message":"Request error, retrying\nGET http://elasticsearch:9200/_xpack => connect ECONNREFUSED 172.21.0.2:9200"}
{"type":"log","@timestamp":"2017-12-31T12:09:11Z","tags":["status","plugin:xpack_main@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch:9200."}
{"type":"log","@timestamp":"2017-12-31T12:09:11Z","tags":["status","plugin:graph@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from red to red - No Living connections","prevState":"red","prevMsg":"Unable to connect to Elasticsearch at http://elasticsearch:9200."}
{"type":"log","@timestamp":"2017-12-31T12:09:19Z","tags":["status","plugin:reporting@6.1.0","error"],"pid":1,"state":"red","message":"Status changed from uninitialized to red - No Living connections","prevState":"uninitialized","prevMsg":"uninitialized"}

12

brainwave...
The official docker elk stack sepb/elk (or something like that) runs even when all my containers are there.. so with very little memory... So I guess it is not just the memory

13

any updates on this one ?

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.