Distributed nodes

Elastic Stack Elasticsearch
1

Hi,

I have an Elasticsearch cluster with 6 nodes on a single system, and it contains some data. Now, I want to add more nodes on different systems. For example, nodes 3-6 will be in VMs 3-6 (each VM should have one data node).

I want to know if Elasticsearch can access data on another system, or if it doesn't matter which node is on which system. I also do not want to store any data in the other VMs, just on the master node VM.

in this case the search latency will reduce or not?

Thank you

2

Can you provide a little more context here? You have 6 instances running on the same machine, or you have a host where you are running 6 VMs? It is not clear.

Also not clear what you mean with access data on another system. If a node is part of an Elasticsearch cluster, then the cluster will get the data in the system it is stored.

If you add a data node, then this is not possible, data nodes will store data, if you do not want you need to configure the node to not be a data node, it will then work as a coordinating node that only receives request and redirect it to other nodes in the cluster, but it does not make any sense to have coordinating nodes in such a small cluster.

Can you provide more context of what you want to do?

2 Likes
3

Sorry for the confusion; I'm a beginner :slight_smile:

I think it would be helpful to include my docker-compose configuration here:

version: "2.2"

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es02\n"\
          "    dns:\n"\
          "      - es02\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es03\n"\
          "    dns:\n"\
          "      - es03\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es04\n"\
          "    dns:\n"\
          "      - es04\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es05\n"\
          "    dns:\n"\
          "      - es05\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es06\n"\
          "    dns:\n"\
          "      - es06\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '

    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    restart: always
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03,es04,es05,es06
      - discovery.seed_hosts=es01,es02,es03,es04,es05,es06
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.remote_cluster_server.ssl.enabled=true
      - xpack.security.remote_cluster_server.ssl.keystore.path=cross-cluster.p12
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - reindex.remote.whitelist=172.19.30.3:9220
      - xpack.license.self_generated.type=${LICENSE}
      - ES_JAVA_OPTS=-Xms100g -Xmx100g
    mem_limit: ${MEM_LIMIT_NODE}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]

      interval: 10s
      timeout: 10s
      retries: 120

  es02:
    depends_on:
      - es01
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata02:/usr/share/elasticsearch/data
    environment:
      - node.name=es02
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03,es04,es05,es06
      - discovery.seed_hosts=es01,es02,es03,es04,es05,es06
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es02/es02.key
      - xpack.security.http.ssl.certificate=certs/es02/es02.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es02/es02.key
      - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
      - reindex.remote.whitelist=192.168.30.5:9200
      - xpack.security.remote_cluster_server.ssl.enabled=true
      - xpack.security.remote_cluster_server.ssl.keystore.path=cross-cluster.p12
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - ES_JAVA_OPTS=-Xms100g -Xmx100g
    mem_limit: ${MEM_LIMIT_NODE}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]

      interval: 10s
      timeout: 10s
      retries: 120

  es03:
    # same as 2

  es04:
    # same as 2,3

  es05:
    # same as 2,3,4
  es06:

    # same as 2,3,4,5
  
  kibana:
    # some configs

  logstash:
    # some configs
  
volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  esdata02:
    driver: local
  esdata03:
    driver: local
  esdata04:
    driver: local
  esdata05:
    driver: local
  esdata06:
    driver: local
  kibanadata:
      driver: local
  logstashdata01:
    driver: local

I have 6 instances running on the same machine. Now i want to split them to another VMs (The VMs are not in the same host)
basically i want to do this to reduce search latency (ignoring network latency, as they are on the same organizational network and not accessible via the internet.)

Currently, when I search my indices, all nodes are on the same machine (VM 1), and the data is also stored in VM 1 (Docker volumes). Now, I want to set up the other nodes on different VMs, which (I think) cannot directly access these volumes to read data( If there’s a solution to this, I’d be happy to know). I’m curious if it’s possible to pass search operations to these nodes if they cannot see the volumes. Can the other nodes simply ingest data into VM 1?

I hope this clarifies things. If there’s anything else I missed, please let me know.

I appreciate your time in reading this and helping me out!

4

I think my bad explanation cause confusions. (sorry for my bad english)

i wanted to distribute nodes to get more I/O rates. And then connect the nodes to master node was my second problem.

I explained here hope to help someone like me :slight_smile: