Dockerized Kibana not able to connect to elasticsearch nodes, unable to retrieve connection

Elastic Stack Kibana
1

Hello,

I am having a complete elastic stack in docker and below is my docker-compose.yml.

version: '3.2'

services:
  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./elasticsearch/config/elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: volume
        source: elasticsearch
        target: /usr/share/elasticsearch/data   
        #Store all ES data from target location to elasticsearch volume
    ports:
      - "9200:9200"
      - "9300:9300"
    container_name: elasticsearch
    environment:
      ES_JAVA_OPTS: "-Xmx2g -Xms2g"
      discovery.type: single-node
    networks:
      - elk

  logstash:
    build:
      context: logstash/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./logstash/config/
        target: /usr/share/logstash/config/
        read_only: true
      - type: bind
        source: ./logstash/pipeline
        target: /usr/share/logstash/pipeline
        read_only: true
    ports:
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    depends_on:
      - elasticsearch

  kibana:
    build:
      context: kibana/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./kibana/config/kibana.yml
        target: /usr/share/kibana/config/kibana.yml
        read_only: true
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - elasticsearch

networks:
  elk:
    driver: bridge

volumes:
  elasticsearch:

docker logs error:

kibana_1         | {"type":"log","@timestamp":"2020-07-17T04:40:40Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1         | {"type":"log","@timestamp":"2020-07-17T04:40:40Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}

Below is my kibana.yml

server.name: kibana
server.host: 0.0.0.0
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.requestTimeout: 120000
monitoring.enabled: false

## X-Pack security credentials
#
elasticsearch.username: elastic
elasticsearch.password: password@123

I check docker network and both the containers are communicating, logged into kibana container and ping ealsticsearch node, able to get response.

As said in few places, tried all options like passing elasticsearch_hosts,elasticsearch_url in docker-compose. Still no luck. The same set-up was working on my local instance. This i am trying in production server. Kindly throw some solution?

2

@Bhanu_Praveen could you try to replace in your docker-compose.yml the following ELASTICSEARCH_URL=http://elasticsearch:9200 by ELASTICSEARCH_HOSTS=http://elasticsearch:9200 ?

Cheers

3

Tried. still same issue..

4

@Bhanu_Praveen could you please try to follow our guide that you can find here https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-docker.html

Probably the only thing you will want to change would be the discovery_type.

Cheers

5

If i comment discovery_type, still the same issue

6

@Bhanu_Praveen could you please double check your configuration with the one suggested in the documentation I linked above? Let me know if after replicating what we suggest you still experience any problem.

Cheers

7

Hi tiago,

I followed above document with initial set-up mentioned. When i try to bring up docker-compose up i see same issue as below:

kib01    | {"type":"log","@timestamp":"2020-07-22T03:03:48Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
kib01    | {"type":"log","@timestamp":"2020-07-22T03:03:50Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://es01:9200/"}
8

I was able to get this working by updating a few things, honestly not sure what it is what did it though.

version: '3.2'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    environment:
      ES_JAVA_OPTS: "-Xmx2g -Xms2g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - type: bind
        source: ./elasticsearch/config/elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: volume
        source: elasticsearch
        target: /usr/share/elasticsearch/data   
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - elastic

  kibana:
    image: docker.elastic.co/kibana/kibana:7.8.0
    volumes:
      - type: bind
        source: ./kibana/config
        target: /usr/share/kibana/config
        read_only: true
    ports:
      - "5601:5601"
    networks:
      - elastic
    depends_on:
      - elasticsearch

networks:
  elastic:
    driver: bridge

volumes:
  elasticsearch:
    driver: local

I removed all the environment settings and moved them to their associated config, since you're mounting those:

kibana/config/kibana.yml:

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200

elasticsearch/config/elasticsearch.yml:

---
## Default Elasticsearch configuration from elasticsearch-docker.
## from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
# Details: https://github.com/elastic/elasticsearch/pull/17288
discovery.zen.minimum_master_nodes: 1

## Use single node discovery in order to disable production mode and avoid bootstrap checks
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
#
discovery.type: single-node

## X-Pack settings
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-xpack.html
#
xpack.license.self_generated.type: trial
1 Like
9

Hello tylers,

In my dev environment, everything working fine. I am trying to bring it up on production server, thats when the connectivity issue occurs. I tried same as you mentioned. Still issue exists. Thanks

10

If it's working for you locally, but not in another environment that leads me to believe it's environment related. Or, you are using a different configuration. What differences are there between what you are doing that is working, to what is not?

11

Hi tylers,

ya, got the issue fixed. Docker is picking IPv6 in production server which is not getting exposed all those ports. Enabled IPv4 and started working fine. Thanks for your time.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.