Dockerized kibana not connecting to elasticsearch in prod server

[Bhanu_Praveen]

Hello,

I had a docker compose set-up of whole elastic stack and the same is working fine on my dev instance. Once i moved the whole to production server, facing connectivity issues between kibana and elastic search.

Please find the details below:

docker-compose.yml

version: '3.7'

services:
  elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./elasticsearch/config/elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: volume
        source: elasticsearch
        target: /usr/share/elasticsearch/data   
    ports:
      - "9200:9200"
      - "9300:9300"
    container_name: elasticsearch
    environment:
      ES_JAVA_OPTS: "-Xmx2g -Xms2g"
      #ES_JAVA_OPTS: "-Des.enforce.bootstrap.checks=true"
      discovery.type: single-node
    networks:
      - elk

  logstash:
    build:
      context: logstash/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./logstash/config/
        target: /usr/share/logstash/config/
        read_only: true
      - type: bind
        source: ./logstash/pipeline
        target: /usr/share/logstash/pipeline
        read_only: true
    ports:
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    depends_on:
      - elasticsearch

  kibana:
    build:
      context: kibana/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./kibana/config/kibana.yml
        target: /usr/share/kibana/config/kibana.yml
        read_only: true
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - elasticsearch

networks:
  elk:
    driver: bridge
      
volumes:
  elasticsearch:

kibana.yml

server.name: kibana
server.host: 0.0.0.0
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.requestTimeout: 120000
elasticsearch.ssl.verificationMode: none

elasticsearch.username: elastic
elasticsearch.password: healthcenter@123

elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0

xpack.license.self_generated.type: basic
xpack.security.enabled: false
xpack.monitoring.collection.enabled: true

docker-compose logs output:

kibana_1         | {"type":"log","@timestamp":"2020-07-21T08:57:31Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1         | {"type":"log","@timestamp":"2020-07-21T08:57:31Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
kibana_1         | {"type":"log","@timestamp":"2020-07-21T08:57:34Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}
kibana_1         | {"type":"log","@timestamp":"2020-07-21T08:57:34Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"No living connections"}
kibana_1         | {"type":"log","@timestamp":"2020-07-21T08:57:36Z","tags":["warning","elasticsearch","admin"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}

Kindly let me know what would be the issue on prod server when same is working on my dev? Note: firewall is disabled on this server

[tsullivan (Tim Sullivan)]

You can try docker-compose exec kibana /bin/bash while the Elasticsearch and Kibana containers are running, to get into a prompt where you can troubleshoot the connection.

~/elastic/esd(master↑1|✔) % docker-compose run kibana /bin/bash
bash-4.2$ ping elasticsearch
PING elasticsearch (172.23.0.4) 56(84) bytes of data.
64 bytes from esd_elasticsearch-internal-7.6.2.esd_default (172.23.0.4): icmp_seq=1 ttl=64 time=0.119 ms
64 bytes from esd_elasticsearch-internal-7.6.2.esd_default (172.23.0.4): icmp_seq=2 ttl=64 time=0.105 ms
^C
--- elasticsearch ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1015ms
rtt min/avg/max/mdev = 0.105/0.112/0.119/0.007 ms

bash-4.2$ curl --cacert config/certificates/ca/ca.crt  https://elasticsearch:9200
{
  "name" : "b4cdfb63f8f1",
  "cluster_name" : "esd",
  "cluster_uuid" : "wHbFZUvDRM2vx7LrGAk69w",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}