Newbie: CentOS + Docker + Elastic Stack = Master not discovered yet

Elastic Stack Elasticsearch
1

I'm trying to start learning Elastic Stack, but I can't get it to even start.

I have:

Nothing else...

curl -X GET "localhost:9200/_cat/nodes?v&pretty" times out and logs are showing

es02 | {"type": "server", "timestamp": "2020-11-12T16:24:53,337Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [es01, es02, es03] to bootstrap a cluster: have discovered [{es02}{t_NDy9i1RlqsgLreV0nvFg}{qX-KZ7raTjKWnSbCkcxCaw}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=8073846784, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [172.18.0.3:9300, 172.18.0.4:9300] from hosts providers and [{es02}{t_NDy9i1RlqsgLreV0nvFg}{qX-KZ7raTjKWnSbCkcxCaw}{172.18.0.2}{172.18.0.2:9300}{cdhilmrstw}{ml.machine_memory=8073846784, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }

I would love to see this running...

What can possible be wrong?

Thank you very much for your support!

2

Should I try 7.9.3 to see if it works better? Or is this an issue within CentOS firewalling or Docker?

Actually I don’t know where to start looking...

3

I’ll give Elastic another chance by installing CentOS 7, Docker, Docker Compose and “Run Elastic Stack on Docker” to see if it’s working.

4

Nope, that didn't do it. Same result.

I can't imagine why a first install according to suggested methods fails.

5

Here is how I'm setting that with docker compose ( docker-compose.yml):

---
version: '3'
services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:$ELASTIC_VERSION
    environment:
      - bootstrap.memory_lock=true
      - discovery.type=single-node
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=$ELASTIC_SECURITY
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - 9200:9200
    networks: ['stack']

  kibana:
    image: docker.elastic.co/kibana/kibana:$ELASTIC_VERSION
    environment:
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=$ELASTIC_PASSWORD
    ports: ['5601:5601']
    networks: ['stack']
    links: ['elasticsearch']
    depends_on: ['elasticsearch']

networks:
  stack: {}

.env file is:

ELASTIC_VERSION=7.10.0
ELASTIC_SECURITY=true
ELASTIC_PASSWORD=changeme
6

Thank you, will try.

How do you refer the .env file?

7

You don't. It's read by default by docker-compose.

8

Thank you very much for your reply and support!

I get a lot of connection errors. How can I troubleshoot the networking issues? Are they within Docker or CentOS?

{"type":"log","@timestamp":"2020-11-13T08:52:56Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"Unable to revive connection: http://elasticsearch:9200/"}

{"type":"log","@timestamp":"2020-11-13T08:52:56Z","tags":["warning","elasticsearch","monitoring"],"pid":6,"message":"No living connections"}

{"type":"log","@timestamp":"2020-11-13T08:52:56Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}

{"type":"log","@timestamp":"2020-11-13T08:52:59Z","tags":["error","elasticsearch","data"],"pid":6,"message":"[ConnectionError]: connect EHOSTUNREACH 172.19.0.2:9200"}

9

Did you just used the config I shared or modified it?

10

Exactly your config. Containers are up and running, and I can authenticate. But they don't see each other. Or... A ping from one container to the other works. But not a curl, and the errors are also as above.

curl elasticsearch -> No route to host
curl http://elasticsearch:9200/ -> No route to host
curl elasticdoesntwork -> Unknown error
curl http://elasticdoesntwork:9200/ -> Unknown error

11

What does a nslookup elasticsearch or nslookup elasticdoesntwork returns from your containers ? What if you are using ip adress instead of it ?

12

Sorry but what do you mean by containers can't see each others? In my example, there's only one elasticsearch node. If you have multiple nodes, it means that you did something else than just running docker-compose up right?

13

You have an elasticsearch container and a kibana container, and the kibana container will have to connect to the elasticsearch container on port 9200.

In my environment, Kibana fails to do so.

14

Are you on to something? nslookup is a command not found neither from the kibana container not the host CentOS 8 Minimal.

15

How did you launch the containers?

16

I have tried:

docker-compose up
docker-compose up -d
Starting from Portainer
docker run (from these examples https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html)

I have tried doing every scenario as root and non-root.

And... It has always been the cannot find host http://elasticsearch:9200/ error. Kibana cannot find it and neither other ES nodes when making a cluster. That, together with curl returning “No route” has always been the problem with CentOS. I didn’t have the problem when running Docker on Ubuntu Server 20.04.

Any ideas?

17

Ok, firewalld process wasn’t automated by Docker, like this article says:

https://medium.com/@Azhagarasu/docker-and-firewalld-mess-in-centos-7-934cb7c7fb48

So, you manually have to set trust to new containers and networks.

Case closed

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.