Unable to revive connection: http://localhost:9200 with Docker

[gcio]

Hi everyone, I'm starting a docker container as follows

docker run -d --rm --name ejq-elk -v $PWD:/var/backups -p 5601:5601 sebp/elk:7.10.0

However, this returns a connection error

==> /var/log/kibana/kibana5.log <==

{"type":"log","@timestamp":"2021-01-11T13:12:42Z","tags":["error","elasticsearch","data"],"pid":338,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

{"type":"log","@timestamp":"2021-01-11T13:12:44Z","tags":["warning","elasticsearch","monitoring"],"pid":338,"message":"Unable to revive connection: http://localhost:9200/"}

{"type":"log","@timestamp":"2021-01-11T13:12:44Z","tags":["warning","elasticsearch","monitoring"],"pid":338,"message":"No living connections"}

{"type":"log","@timestamp":"2021-01-11T13:12:44Z","tags":["warning","plugins","licensing"],"pid":338,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}

{"type":"log","@timestamp":"2021-01-11T13:12:44Z","tags":["warning","elasticsearch","data"],"pid":338,"message":"Unable to revive connection: http://localhost:9200/"}

{"type":"log","@timestamp":"2021-01-11T13:12:44Z","tags":["warning","elasticsearch","data"],"pid":338,"message":"No living connections"}

{"type":"log","@timestamp":"2021-01-11T13:12:44Z","tags":["warning","plugins","licensing"],"pid":338,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}

{"type":"log","@timestamp":"2021-01-11T13:12:44Z","tags":["warning","elasticsearch","data"],"pid":338,"message":"Unable to revive connection: http://localhost:9200/"}

{"type":"log","@timestamp":"2021-01-11T13:12:44Z","tags":["warning","elasticsearch","data"],"pid":338,"message":"No living connections"}

{"type":"log","@timestamp":"2021-01-11T13:12:44Z","tags":["error","plugins","taskManager","taskManager"],"pid":338,"message":"Failed to poll for work: Error: No Living connections"}

{"type":"log","@timestamp":"2021-01-11T13:12:44Z","tags":["error","elasticsearch","data"],"pid":338,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

{"type":"log","@timestamp":"2021-01-11T13:12:44Z","tags":["error","elasticsearch","data"],"pid":338,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

Does anyone know why this might happen?

Thank you in advance

[majagrubic (Maja)]

Is your ES cluster up and running?

[gcio]

Hi Maja, thank you for your answer.

I think so:

 * Starting periodic command scheduler cron
   ...done.
 * Starting Elasticsearch Server
   ...done.
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
Waiting for Elasticsearch cluster to respond (1/30)
logstash started.
 * Starting Kibana5
   ...done.

Is this what you mean?
G

[gcio]

This is another part of the log, not sure if it helps

==> /var/log/kibana/kibana5.log <==
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"[ConnectionError]: write EPIPE"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"[ConnectionError]: socket hang up"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"Request error, retrying\nPUT http://localhost:9200/.apm-custom-link => socket hang up"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"Request error, retrying\nPUT http://localhost:9200/.apm-agent-configuration => socket hang up"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"[ConnectionError]: socket hang up"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["warning","plugins","telemetry"],"pid":330,"message":"Unable to update legacy telemetry configs."}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"[ConnectionError]: socket hang up"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"[ConnectionError]: socket hang up"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"[ConnectionError]: socket hang up"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"[ConnectionError]: socket hang up"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["warning","plugins","apm"],"pid":330,"message":"Failed to fetch saved telemetry data."}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"[ConnectionError]: socket hang up"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["warning","environment"],"pid":330,"message":"Detected an unhandled Promise rejection.\nConnectionError: socket hang up"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"Request error, retrying\nGET http://localhost:9200/_ilm/policy/kibana-event-log-policy => socket hang up"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["error","elasticsearch","data"],"pid":330,"message":"Request error, retrying\nPOST http://localhost:9200/.kibana_task_manager/_update_by_query?ignore_unavailable=true&refresh=true&max_docs=10&conflicts=proceed => connect ECONNREFUSED 127.0.0.1:9200"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["warning","elasticsearch","data"],"pid":330,"message":"Unable to revive connection: http://localhost:9200/"}
{"type":"log","@timestamp":"2021-01-11T13:36:39Z","tags":["warning","elasticsearch","data"],"pid":330,"message":"No living connections"}

[majagrubic (Maja)]

The error you're receiving is because Kibana cannot establish connection to Elasticsearch. Possible root cause is that something is wrong in Kibana configuration or that the Elasticsearch cluster didn't start properly. Could you check the health of ES cluster?
curl -X GET "localhost:9200/_cluster/health?pretty"

[gcio]

Here you go. It seems that it cannot connect to 9200 at all.

(base) $:~ curl -X GET "localhost:9200/_cluster/health?pretty"
curl: (7) Failed to connect to localhost port 9200: Connection refused

[majagrubic (Maja)]

Just to be clear: are you running the Docker Elasticsearch image at all? You should have two Docker containers running in parallel: one for Elasticsearch and for Kibana. Or did you just run Kibana image without running Elasticsearch as well?

[gcio]

Ehm, no

OK, trying now following this https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

I'll let you know how it goes. Thank you so much.
G

[Jurilz]

If your Kibana is running in a docker container, than for this Kibana instance "localhost" is not your machine (which you are forwarding the 9200 port to) but it's container, and in the Kibana-Docker-Container there is no Elasticsearch under localhost:9200.

In this case you have to put both containers in the same (docker)-network and address elasticsearch like this name-of-elasticsarch-container:9200 from the Kibana config.

[gcio]

Thank you Maja and Jurilz, it works now!

All the best,
G