Load balance in elasticsearch with multiple nodes in same machine


(Suraj) #1

Hi,

I am trying to run multiple nodes with different instances in same machine for load balancing in elasticsearch.

I have downloaded three elasticsearch files in the same machine and configured their elasticsearch.yml file with different settings. Please see below:

  1. Elasticsearch - default configuration

  2. Elasticsearch-node1
    cluster.name: Animals
    node.name: Snake
    node.master: true
    node.data: true
    transport.tcp.port: 9300
    http.port: 9200
    discovery.zen.ping.unicast.hosts: ["Snake", "Baboon"]

  3. Elasticsearch-node2
    cluster.name: Animals
    node.name: Baboon
    node.master: false
    node.data: true
    transport.tcp.port: 9301
    http.port: 9201
    discovery.zen.ping.unicast.hosts: ["Snake", "Baboon"]

Please share your opinion and ideas on this.

Thank you.


#2

You don't really need three different binaries (at least on Linux). You can start separate processes with different config files and data paths...

I would leave all as node.master: true. You are also using both for discovery.zen.ping.unicast.hosts:

Default is ports 9200/9300 so node1 (your second node) can't use those, bump them up with one for second and third node.

I'm running 5 nodes on same machine on 6 machines in total over two clusters. Definitely can be done :slight_smile:


(Suraj) #3

Hi,

Thanks for the response.

I am able to create multiple instances with the nodes.

Please find my elasticsearch.yml config below.

  • Elasticsearch - default configuration

  • Elasticsearch-node1
    cluster.name: Animals
    node.name: Snake
    node.master: true
    node.data: true
    path.data: "C:\elasticsearch-5.4.2-node1\elasticsearch-5.4.2\data"
    path.logs: "C:\elasticsearch-5.4.2-node1\elasticsearch-5.4.2\logs"
    http.port: 9201
    discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

  • Elasticsearch-node2
    cluster.name: Animals
    node.name: Baboon
    node.master: false
    node.data: true
    path.data: "C:\elasticsearch-5.4.2-node2\elasticsearch-5.4.2\data"
    path.logs: "C:\elasticsearch-5.4.2-node2\elasticsearch-5.4.2\logs"
    http.port: 9202
    discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

FYI, the status is red in the cluster health.

And I am using logstash to load the csv file into elasticsearch for which I have used the below config. But I am not able to see the other instances(9201 and 9202) loading the data.

output
{
elasticsearch {
hosts => ["localhost:9200", "localhost:9201", "localhost:9202"]
index => "cmss"
document_type => "cms"
document_id => "%{id}"
}
}

Thank you.


(Suraj) #4

Hi,

Thanks for the response.

I am able to create multiple instances with the nodes.

Please find my elasticsearch.yml config below.

Elasticsearch - default configuration

Elasticsearch-node1
cluster.name: Animals
node.name: Snake
node.master: true
node.data: true
path.data: "C:\elasticsearch-5.4.2-node1\elasticsearch-5.4.2\data"
path.logs: "C:\elasticsearch-5.4.2-node1\elasticsearch-5.4.2\logs"
http.port: 9201
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

Elasticsearch-node2
cluster.name: Animals
node.name: Baboon
node.master: false
node.data: true
path.data: "C:\elasticsearch-5.4.2-node2\elasticsearch-5.4.2\data"
path.logs: "C:\elasticsearch-5.4.2-node2\elasticsearch-5.4.2\logs"
http.port: 9202
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

FYI, the status is red in the cluster health.
And I am using logstash to load the csv file into elasticsearch for which I have used the below config. But I am not able to see the other instances(9201 and 9202) loading the data.

output
{
elasticsearch {
hosts => ["localhost:9200", "localhost:9201", "localhost:9202"]
index => "cmss"
document_type => "cms"
document_id => "%{id}"
}
}

Thank you.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.