Trying to setup multi-node Elasticsearch cluster environment

[milindyedge]

I am new to Elasticsearch and currently trying to setup a multi-node Elasticsearch cluster environment using docker in my local machine. When cluster comes up, it status is green (checked using "/_cluster/health" api, however it has only one node in it whereas I am trying to create 3 nodes cluster. Below is the docker-compose.yml for the same. Note that in below docker-compose file, I am mapping the index volume to each node using unique directories and expecting that when 3 node cluster come sup, this index gets distributed across shards on multiple nodes.

version: '3.3'

services:

  elasticsearch01:
    image: dockerregistry.homeoffice.anfcorp.com/vendor/elasticsearch/elasticsearch:7.8.0
    ports:
      - "9200:9200"
    environment:
      cluster.name: "docker-cluster"
      node.name: "elasticsearch01"
      discovery.seed_hosts: "elasticsearch02"
      network.host: 0.0.0.0
      indices.query.bool.max_clause_count: 100000
      transport.tcp.port: 9300
      cluster.initial_master_nodes: "elasticsearch01,elasticsearch02"
    volumes:
      - ./volumes/elasticsearch01/data:/usr/share/elasticsearch/data
      
  elasticsearch02:
    image: dockerregistry.homeoffice.anfcorp.com/vendor/elasticsearch/elasticsearch:7.8.0
    ports:
      - "9201:9200"
    environment:
      cluster.name: "docker-cluster"
      node.name: "elasticsearch02"
      network.host: 0.0.0.0
      indices.query.bool.max_clause_count: 100000
      discovery.seed_hosts: "elasticsearch01"
      transport.tcp.port: 9300
      cluster.initial_master_nodes: "elasticsearch01,elasticsearch02"
    volumes:
      - ./volumes/elasticsearch02/data:/usr/share/elasticsearch/data

  elasticsearch03:
    image: dockerregistry.homeoffice.anfcorp.com/vendor/elasticsearch/elasticsearch:7.8.0
    ports:
      - "9202:9200"
    environment:
      cluster.name: "docker-cluster"
      node.name: "elasticsearch03"
      network.host: 0.0.0.0
      indices.query.bool.max_clause_count: 100000
      discovery.seed_hosts: "elasticsearch01"
      transport.tcp.port: 9300
      cluster.initial_master_nodes: "elasticsearch01,elasticsearch02"
    volumes:
      - ./volumes/elasticsearch03/data:/usr/share/elasticsearch/data
    ```

[warkolm (Mark Walkom)]

Welcome to our community!

Take a look at Install Elasticsearch with Docker | Elasticsearch Guide [7.15] | Elastic as an example of this I would suggest that you not run 7.8 as it's EOL.

[milindyedge]

Hi warkolm

I tried using the docker-compose file from the link you mentioned above. docker-compose.yml mentioned in here works as is and I can check cluster health using api which is green and it has 3 nodes on it. However, I see some errors when doing docker-compose up. Also when cluster comes up, it does not have my index on it (I have created data01, data02, data03 directories at the level of docker-compose.yml file and each of these directories have node directory in it which has the index I downloaded from another environment). I am expecting this index to be available when multi node cluster environment comes up.
Using below APIs to check cluster health and query index -
http://0.0.0.0:9200/_cluster/health

Output

"cluster_name": "es-docker-cluster",
"status": "green",
"timed_out": false,
"number_of_nodes": 3,
"number_of_data_nodes": 3,
"active_primary_shards": 1,
"active_shards": 2,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100
}```

API to check index - `http://0.0.0.0:9200/_search`
Output - 

{
"took": 4,
"timed_out": false,
"_shards": {
"total": 0,
"successful": 0,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": 0,
"hits":
}
}

Error when doing docker-compose up

{"type": "server", "timestamp": "2021-10-27T13:46:40,794Z", "level": "WARN", "component": "o.e.c.s.MasterService", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "failing [elected-as-master ([2] nodes joined)[{es03}{KjTx5A_VTRyRUHuDsc5HjA}{4_LRIYm0RJChWNCcIfLZyA}{172.18.0.2}{172.18.0.2:9300}{cdfhilmrstw} elect leader, {es01}{FVWRNW9eQA2haHql0Vucvg}{eFMazF0QSrGbXMwxNeKc_Q}{172.18.0.4}{172.18.0.4:9300}{cdfhilmrstw} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION]]: failed to commit cluster state version [1]",
es01 | "stacktrace": ["org.Elasticsearch.cluster.coordination.FailedToCommitClusterStateException: node is no longer master for term 1 while handling publication",
es01 | "at org.Elasticsearch.cluster.coordination.Coordinator.publish(Coordinator.java:1074) ~[Elasticsearch-7.15.1.jar:7.15.1]",
es01 | "at org.Elasticsearch.cluster.service.MasterService.publish(MasterService.java:272) [Elasticsearch-7.15.1.jar:7.15.1]",
es01 | "at org.Elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:254) [Elasticsearch-7.15.1.jar:7.15.1]",
es01 | "at org.Elasticsearch.cluster.service.MasterService.access$000(MasterService.java:63) [Elasticsearch-7.15.1.jar:7.15.1]",
es01 | "at org.Elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:155) [Elasticsearch-7.15.1.jar:7.15.1]",
es01 | "at org.Elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:139) [Elasticsearch-7.15.1.jar:7.15.1]",
es01 | "at org.Elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:177) [Elasticsearch-7.15.1.jar:7.15.1]",
es01 | "at org.Elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [Elasticsearch-7.15.1.jar:7.15.1]",
es01 | "at org.Elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:259) [Elasticsearch-7.15.1.jar:7.15.1]",
es01 | "at org.Elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:222) [Elasticsearch-7.15.1.jar:7.15.1]",
es01 | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
es01 | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
es01 | "at java.lang.Thread.run(Thread.java:833) [?:?]"] }