Unable to run elastic stack on docker

[y0y0z]

i create a specification for the stack using docker-compose:

# docker-compose.yml
version: '3.7'
services:
  elasticsearch:
    container_name: elasticsearch
    image: elasticsearch:7.6.0
    ports:
      - "9200:9200"
    environment:
      discovery.type: single-node

  kibana:
    container_name: kibana
    image: kibana:7.6.0
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
    environment:
      ELASTICSEARCH_HOSTS: http://elasticsearch:9200

then, i spin up the containers

$ docker-compose up -d
Creating network "tmp_default" with the default driver
Creating elasticsearch ... done
Creating kibana        ... done

checking the health of the elasticsearch from my host operating system:

$ curl -s 'localhost:9200/_cluster/health?wait_for_status=yellow&timeout=50s' | jq
{
  "cluster_name": "docker-cluster",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "active_primary_shards": 3,
  "active_shards": 3,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

verifying that elasticsearch is reachable from kibana container:

$ docker-compose exec -u root kibana curl -s 'elasticsearch:9200/_cluster/health?wait_for_status=yellow&timeout=50s' | jq
{
  "cluster_name": "docker-cluster",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "active_primary_shards": 3,
  "active_shards": 3,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

verifying that the environment variables in the docker-compose file took precedence over the kibana.yml configuration file:

$ docker-compose exec kibana ps -efww | grep node
kibana       6     1  4 22:08 ?        00:00:57 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli --cpu.cgroup.path.override=/ --cpuacct.cgroup.path.override=/ --elasticsearch.hosts=http://elasticsearch:9200

verifying that kibana is reachable from elasticsearch container:

$ docker-compose exec elasticsearch ping kibana -c 1
PING kibana (192.168.224.3) 56(84) bytes of data.
64 bytes from kibana.tmp_default (192.168.224.3): icmp_seq=1 ttl=64 time=0.342 ms

--- kibana ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.342/0.342/0.342/0.000 ms

double verify that the elasticsearch and kibana are on the same network:

$ docker network inspect tmp_default --format '{{json .}}' | jq '.Containers'
{
  "c4378048f314429d47fe145ec426d992fb19b0843a205d35cd99be96b0202946": {
    "Name": "elasticsearch",
    "EndpointID": "67631e0128bae3e2f7c3aae7bb504672e685678c9f259ddd73d652b654790e24",
    "MacAddress": "02:42:c0:a8:e0:02",
    "IPv4Address": "192.168.224.2/20",
    "IPv6Address": ""
  },
  "ea5d77068a1c0c26d30cb5268ed94be6bbb0c50332e104245d49a56c0e45d78f": {
    "Name": "kibana",
    "EndpointID": "d444344d5d8fef2f5e67dd74ec9c97a1b49b3bc80f1c6b117bcf8ceb6bbcf952",
    "MacAddress": "02:42:c0:a8:e0:03",
    "IPv4Address": "192.168.224.3/20",
    "IPv6Address": ""
  }
}

navigating to kibana management, there is no indication for tje connection to elasticsearch cluster that was specified in the docker-compose.yml.

wen adding an elasticsearch remote cluster manually, by typing elasticsearch:9200 in the seed nodes field, then click save - it fails, but succeeds for elasticsearch:9300. why?

replacing in docker-compose.yml the line ELASTICSEARCH_HOSTS: http://elasticsearch:9200 with ELASTICSEARCH_HOSTS: http://elasticsearch:9300 and repeat everything, didn't resolve the issue.

i even tried another docker-compose.yml file, but got the same behavior. would appreciate if you could assist me making the elastic stack working.

[spinscale (Alexander Reelsen)]

quick guess from reading your docker setup: By default elasticsearch only listens on localhost, so it is not exposed to the outside.

try

 environment:
      - http.host=0.0.0.0

for the elasticsearch service to make sure that the HTTP service is exposed to the outside.

[y0y0z]

@spinscale: yet, elasticsearch is accessible from kibana. please see above. anyhow, gonna try your offering and will report back in a few.

[y0y0z]

@spinscale: your offer did not change anything.

also, wanted to verify your claim. so using the docker-compose.yml i posted, i started the elasticsearch and then:

$ docker-compose exec elasticsearch /bin/bash -c 'yum update -q && yum install net-tools -y -q && netstat -nlupt'
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.11:34719        0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:9200            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:9300            0.0.0.0:*               LISTEN      -
udp        0      0 127.0.0.11:42996        0.0.0.0:*                           - 

as you can see, elasticsearch do listen to 0.0.0.0

[y0y0z]

after debugging using the command line, i can see kibana indecies on elasticsearch but see no indication for the elasticsearch connection on kibana's gui.

feels like a gui bug.