Unable to create elasticsearch cluster in docker swarm

sammi · March 29, 2020, 5:22pm

i am trying to create a 2 node docker swarm elasticsearch cluster. have been trying this from 2 days but couldn't succeed. please help me. below is the compose file, first node joins the cluster but second not able to join. When i run the 2 nodes in same cluster 2 nodes becoming cluster but getting below error on second node when i run them in 2 separate host docker nodes.

bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed.

Below is the compose file, any suggestions would be great help!
""""
version: '3.7'
services:
es01:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
container_name: es01
environment:
- node.name=es01
- cluster.name=es-docker-cluster
- bootstrap.memory_lock=false
- network.host=0.0.0.0
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- discovery.zen.ping.unicast.hosts=es01,es02
- discovery.zen.minimum_master_nodes=1
- node.master=true
- node.data=true
- node.ingest=true
volumes:
- /opt/es01:/usr/share/elasticsearch/data
ports:
- 9200:9200
deploy:
placement:
constraints:
- "node.hostname == node1"
es02:
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
container_name: es02
environment:
- node.name=es02
- cluster.name=es-docker-cluster
- bootstrap.memory_lock=false
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- discovery.zen.ping.unicast.hosts=es01,es02
- discovery.zen.minimum_master_nodes=1
- node.master=false
- node.data=true
- node.ingest=true
volumes:
- /opt/es02:/usr/share/elasticsearch/data
deploy:
placement:
constraints:
- "node.hostname == node2"
""""

dadoonet · March 29, 2020, 6:35pm

What is the rest of the logs?

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

dadoonet · March 30, 2020, 8:22am

Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

sammi · March 30, 2020, 8:27am

Below is the docker compose file which i am using deploy it as swarm.

version: '3.7'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - bootstrap.memory_lock=false
      - network.host=0.0.0.0
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.zen.ping.unicast.hosts=es01,es02
      - discovery.zen.minimum_master_nodes=1
      - node.master=true
      - node.data=true
      - node.ingest=true
    volumes:
      - /opt/es01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    deploy:
      placement:
        constraints:
          - "node.hostname == node1"
  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - bootstrap.memory_lock=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.zen.ping.unicast.hosts=es01,es02
      - discovery.zen.minimum_master_nodes=1
      - node.master=false
      - node.data=true
      - node.ingest=true
    volumes:
      - /opt/es02:/usr/share/elasticsearch/data
    deploy:
      placement:
        constraints:
          - "node.hostname == node2"

Here are the logs from node1:

{"type": "server", "timestamp": "2020-03-30T06:30:40,328Z", "level": "INFO", "component": "o.e.c.s.MasterService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "elected-as-master ([1] nodes joined)[{es01}{TxAWx5BJRKWaFA4_n7dIDA}{OlRlsCl9SrCT9qNDlbIjSQ}{10.0.61.3}{10.0.61.3:9300}{dilm}{ml.machine_memory=135024914432, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 9, version: 40, delta: master node changed {previous [], current [{es01}{TxAWx5BJRKWaFA4_n7dIDA}{OlRlsCl9SrCT9qNDlbIjSQ}{10.0.61.3}{10.0.61.3:9300}{dilm}{ml.machine_memory=135024914432, xpack.installed=true, ml.max_open_jobs=20}]}" }
{"type": "server", "timestamp": "2020-03-30T06:30:40,377Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "master node changed {previous [], current [{es01}{TxAWx5BJRKWaFA4_n7dIDA}{OlRlsCl9SrCT9qNDlbIjSQ}{10.0.61.3}{10.0.61.3:9300}{dilm}{ml.machine_memory=135024914432, xpack.installed=true, ml.max_open_jobs=20}]}, term: 9, version: 40, reason: Publication{term=9, version=40}" }
{"type": "server", "timestamp": "2020-03-30T06:30:40,418Z", "level": "INFO", "component": "o.e.h.AbstractHttpServerTransport", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "publish_address {10.0.61.3:9200}, bound_addresses {0.0.0.0:9200}", "cluster.uuid": "_s5dqoIUSAyNPNUvb-TeBg", "node.id": "TxAWx5BJRKWaFA4_n7dIDA"  }
{"type": "server", "timestamp": "2020-03-30T06:30:40,418Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "started", "cluster.uuid": "_s5dqoIUSAyNPNUvb-TeBg", "node.id": "TxAWx5BJRKWaFA4_n7dIDA"  }
{"type": "server", "timestamp": "2020-03-30T06:30:40,627Z", "level": "INFO", "component": "o.e.l.LicenseService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "license [2db2e78c-fd6f-4c49-b4a6-a230739bac5c] mode [basic] - valid", "cluster.uuid": "_s5dqoIUSAyNPNUvb-TeBg", "node.id": "TxAWx5BJRKWaFA4_n7dIDA"  }
{"type": "server", "timestamp": "2020-03-30T06:30:40,628Z", "level": "INFO", "component": "o.e.x.s.s.SecurityStatusChangeListener", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "Active license is now [BASIC]; Security is disabled", "cluster.uuid": "_s5dqoIUSAyNPNUvb-TeBg", "node.id": "TxAWx5BJRKWaFA4_n7dIDA"  }
{"type": "server", "timestamp": "2020-03-30T06:30:40,637Z", "level": "INFO", "component": "o.e.g.GatewayService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "recovered [0] indices into cluster_state", "cluster.uuid": "_s5dqoIUSAyNPNUvb-TeBg", "node.id": "TxAWx5BJRKWaFA4_n7dIDA"  }

logs from node 2:

{"type": "server", "timestamp": "2020-03-30T06:29:00,459Z", "level": "DEBUG", "component": "o.e.a.ActionModule", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "Using REST wrapper from plugin org.elasticsearch.xpack.security.Security" }
{"type": "server", "timestamp": "2020-03-30T06:29:00,577Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "using discovery type [zen] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2020-03-30T06:29:01,403Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "initialized" }
{"type": "server", "timestamp": "2020-03-30T06:29:01,403Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "starting ..." }
{"type": "server", "timestamp": "2020-03-30T06:29:01,525Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "publish_address {10.0.60.12:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2020-03-30T06:29:01,789Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
ERROR: [2] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/es-docker-cluster.log
{"type": "server", "timestamp": "2020-03-30T06:29:01,827Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "stopping ..." }
{"type": "server", "timestamp": "2020-03-30T06:29:01,848Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "stopped" }
{"type": "server", "timestamp": "2020-03-30T06:29:01,848Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "closing ..." }
{"type": "server", "timestamp": "2020-03-30T06:29:01,864Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "closed" }
{"type": "server", "timestamp": "2020-03-30T06:29:01,881Z", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "Native controller process has stopped - no new native processes can be started" }

dadoonet · March 30, 2020, 8:39am

Please read https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html

Christian_Dahlqvist · March 30, 2020, 8:47am

It also looks like you are using old cluster configuration setting that are no longer applicable in Elasticsearch 7.x, e.g. minimum_naster_nodes. Check the docs for examples of how to properly set up a Elasticsearch 7.x cluster.

sammi · March 30, 2020, 8:58am

Thanks for bringing this link up. have gone through this page couple of times from previous posts but still couldn't figure out the problem.
seems to be an issue with with network.host value. tried with all values suggested like site,local, globe and IP of the first container host nothing worked.

can you suggest what should be given for network.host for the second node to join the cluster.

sammi · March 30, 2020, 9:02am

I have tried with elasticsearch 6.8.4 version and 7.6.1 version both have different environment set values where they are not applicable in recent versions. I noticed some observations.

`discovery.seed_hosts and discovery.zen.ping.unicast.hosts can't be used as same time

Even i tried commenting out minimum_master_nodes , still getting same error.

sammi · March 30, 2020, 9:05am

just a note. The same compose file working and 2 nodes joining the cluster when i start the swarm in same node.

i verified it with below command, I can see 2 nodes.

curl http://127.0.0.1:9200/_cluster/health?pretty

It is failing when i start 2 nodes in 2 seperate docker host servers.

sammi · March 30, 2020, 4:23pm

can any one please help on this.

system · April 27, 2020, 4:23pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Can't create new 3 node cluster using docker swarm Elasticsearch docker	1	355	April 29, 2022
Unable to form an ES cluster Elasticsearch docker	2	318	June 1, 2023
Elastic Cluster Fails on Docker Swarm Elasticsearch	1	595	June 10, 2020
Docker Swarm and Elasticsearch Elasticsearch	2	6813	July 15, 2019
Elasticsearch Cluster 7.5.1 Not Working on docker Elasticsearch docker	15	2463	January 28, 2020

Unable to create elasticsearch cluster in docker swarm

Related Topics