# Distributed nodes

**URL:** <https://discuss.elastic.co/t/distributed-nodes/370256>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [November 9, 2024, 2:10pm UTC](https://discuss.elastic.co/t/distributed-nodes/370256 "2024-11-09T14:10:03Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sandra\_a](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandra_a/32/126560_2.png) [@sandra\_a](https://discuss.elastic.co/u/sandra_a)\
**Post date:** [November 9, 2024, 2:10pm UTC](https://discuss.elastic.co/t/distributed-nodes/370256/1 "2024-11-09T14:10:03Z")

</div>

Hi,

I have an Elasticsearch cluster with 6 nodes on a single system, and it contains some data. Now, I want to add more nodes on different systems. For example, nodes 3-6 will be in VMs 3-6 (each VM should have one data node).

I want to know if Elasticsearch can access data on another system, or if it doesn't matter which node is on which system. I also do not want to store any data in the other VMs, just on the master node VM.

in this case the search latency will reduce or not?

Thank you

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [November 9, 2024, 2:26pm UTC](https://discuss.elastic.co/t/distributed-nodes/370256/2 "2024-11-09T14:26:50Z")

</div>

> [@sandra\_a](#):
>
> I have an Elasticsearch cluster with 6 nodes on a single system

Can you provide a little more context here? You have 6 instances running on the same machine, or you have a host where you are running 6 VMs? It is not clear.

> [@sandra\_a](#):
>
> I want to know if Elasticsearch can access data on another system, or if it doesn't matter which node is on which system.

Also not clear what you mean with access data on another system. If a node is part of an Elasticsearch cluster, then the cluster will get the data in the system it is stored.

> [@sandra\_a](#):
>
> I also do not want to store any data in the other VMs, just on the master node VM.

If you add a data node, then this is not possible, data nodes will store data, if you do not want you need to configure the node to not be a data node, it will then work as a coordinating node that only receives request and redirect it to other nodes in the cluster, but it does not make any sense to have coordinating nodes in such a small cluster.

Can you provide more context of what you want to do?

---

<div class="post-metadata">

**Author:** ![sandra\_a](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandra_a/32/126560_2.png) [@sandra\_a](https://discuss.elastic.co/u/sandra_a)\
**Post date:** [November 9, 2024, 3:14pm UTC](https://discuss.elastic.co/t/distributed-nodes/370256/3 "2024-11-09T15:14:02Z")

</div>

Sorry for the confusion; I'm a beginner 🙂

I think it would be helpful to include my `docker-compose` configuration here:

```auto
version: "2.2"

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [x${ELASTIC_PASSWORD} == x]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [x${KIBANA_PASSWORD} == x]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [! -f config/certs/ca.zip]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [! -f config/certs/certs.zip]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          " - name: es01\n"\
          " dns:\n"\
          " - es01\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          " - name: es02\n"\
          " dns:\n"\
          " - es02\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          " - name: es03\n"\
          " dns:\n"\
          " - es03\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          " - name: es04\n"\
          " dns:\n"\
          " - es04\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          " - name: es05\n"\
          " dns:\n"\
          " - es05\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          " - name: es06\n"\
          " dns:\n"\
          " - es06\n"\
          " - localhost\n"\
          " ip:\n"\
          " - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '

    healthcheck:
      test: ["CMD-SHELL", "[-f config/certs/es01/es01.crt]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    restart: always
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03,es04,es05,es06
      - discovery.seed_hosts=es01,es02,es03,es04,es05,es06
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.remote_cluster_server.ssl.enabled=true
      - xpack.security.remote_cluster_server.ssl.keystore.path=cross-cluster.p12
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - reindex.remote.whitelist=172.19.30.3:9220
      - xpack.license.self_generated.type=${LICENSE}
      - ES_JAVA_OPTS=-Xms100g -Xmx100g
    mem_limit: ${MEM_LIMIT_NODE}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]

      interval: 10s
      timeout: 10s
      retries: 120

  es02:
    depends_on:
      - es01
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata02:/usr/share/elasticsearch/data
    environment:
      - node.name=es02
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03,es04,es05,es06
      - discovery.seed_hosts=es01,es02,es03,es04,es05,es06
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es02/es02.key
      - xpack.security.http.ssl.certificate=certs/es02/es02.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es02/es02.key
      - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
      - reindex.remote.whitelist=192.168.30.5:9200
      - xpack.security.remote_cluster_server.ssl.enabled=true
      - xpack.security.remote_cluster_server.ssl.keystore.path=cross-cluster.p12
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - ES_JAVA_OPTS=-Xms100g -Xmx100g
    mem_limit: ${MEM_LIMIT_NODE}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]

      interval: 10s
      timeout: 10s
      retries: 120

  es03:
    # same as 2

  es04:
    # same as 2,3

  es05:
    # same as 2,3,4
  es06:

    # same as 2,3,4,5
  
  kibana:
    # some configs

  logstash:
    # some configs
  
volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  esdata02:
    driver: local
  esdata03:
    driver: local
  esdata04:
    driver: local
  esdata05:
    driver: local
  esdata06:
    driver: local
  kibanadata:
      driver: local
  logstashdata01:
    driver: local

```

> [@leandrojmp](#):
>
> Can you provide a little more context here? You have 6 instances running on the same machine, or you have a host where you are running 6 VMs? It is not clear.

I have 6 instances running on the same machine. Now i want to split them to another VMs (The VMs are not in the same host)  
basically i want to do this to reduce search latency (ignoring network latency, as they are on the same organizational network and not accessible via the internet.)

> [@leandrojmp](#):
>
> Also not clear what you mean with access data on another system. If a node is part of an Elasticsearch cluster, then the cluster will get the data in the system it is stored.

Currently, when I search my indices, all nodes are on the same machine (VM 1), and the data is also stored in VM 1 (Docker volumes). Now, I want to set up the other nodes on different VMs, which (I think) cannot directly access these volumes to read data( If there’s a solution to this, I’d be happy to know). I’m curious if it’s possible to pass search operations to these nodes if they cannot see the volumes. Can the other nodes simply ingest data into VM 1?

> [@leandrojmp](#):
>
> Can you provide more context of what you want to do?

I hope this clarifies things. If there’s anything else I missed, please let me know.

I appreciate your time in reading this and helping me out!

---

<div class="post-metadata">

**Author:** ![sandra\_a](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandra_a/32/126560_2.png) [@sandra\_a](https://discuss.elastic.co/u/sandra_a)\
**Post date:** [November 20, 2024, 7:14am UTC](https://discuss.elastic.co/t/distributed-nodes/370256/4 "2024-11-20T07:14:10Z")

</div>

I think my bad explanation cause confusions. (sorry for my bad english)

i wanted to distribute nodes to get more I/O rates. And then connect the nodes to master node was my second problem.

I explained here hope to help someone like me 🙂

> [@Remote node](https://discuss.elastic.co/t/remote-node/370287/2):
>
> OK. For those who don't know so much like me slight_smile in Host-1 which is your master node host, You should up a cluster with just one master node and if you want use kibana. and on the other Hosts You should setup a single node cluster and change seed\_host and initial\_master\_node to your master node host. then if everything ok you can see the joining message in master node logs and data node logs. for eg.: master node: version: "2.2" services: es01: image: docker.elastic.co/e…