Help with MultiNode communication


(Tellvideo) #1

I have used ELK stack on a single box earlier without any issues.
The scenario i am trying is with 2 machines(my laptop and a virtual machine) with windows os's and ELK modules installed.

I am trying to setup a 4 node cluster using the 2 machines with below configurations.

Nodes in VM
MasterNode-1

cluster.name: ELK_MultiNode
node.name: "multinode_14_master01"
node.master: true
node.data: false
bootstrap.mlockall: true
network.publish_host: 100.201.218.18
network.host: 100.201.218.18
transport.tcp.port: 9300
http.port: 9201
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.timeout: 30s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["100.201.218.18:9300","100.201.218.19:9305"]
http.cors.enabled: true

ESClientNode

cluster.name: ELK_MultiNode
node.name: "multinode_14_client01"
node.master: false
node.data: false
bootstrap.mlockall: true
network.publish_host: 100.201.218.18
network.host: 100.201.218.18
transport.tcp.port: 9302
http.port: 9203
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.timeout: 30s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["100.201.218.19:9300","100.201.218.19:9305"]

Nodes in Laptop
MasterNode-2

cluster.name: ELK_MultiNode
node.name: "multinode_14_master02"
node.master: true
node.data: false
bootstrap.mlockall: true
network.publish_host: 100.201.218.19
network.host: 100.201.218.19
transport.tcp.port: 9305
http.port: 9202
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.timeout: 30s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["100.201.218.18:9300","100.201.218.19:9305"]
http.cors.enabled: true

DataNode

cluster.name: ELK_MultiNode
node.name: "multinode_14_master02"
node.master: false
node.data: true
bootstrap.mlockall: true
index.number_of_shards: 5
index.number_of_replicas: 1
network.publish_host: 100.201.218.19
network.host: 100.201.218.19
transport.tcp.port: 9306
http.port: 9208
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.timeout: 30s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["100.201.218.18:9300","100.201.218.19:9305"]
http.cors.enabled: true

I ran the nodes individually and they all seem to communicate with each other and electing a master on which master node was started first.
There are couple of questions

  1. Are the config entries proper especially discovery.zen.ping.unicast.hosts bcoz i am mentioning both master node IP's here.

  2. I want to index some IIS logs and send my indexing reqquests to the ESClientNode so in the logstash config section below what should be the host (ClientNode IP or the datanode IP)
    because when i give client ip and http port nothing seems to happen.

    output {
    stdout { codec => rubydebug }
    elasticsearch {
    host => ["100.201.218.19"]
    port => "9203"
    protocol => "http"
    index => "logstash-iislog-%{+YYYY.MM.dd}"
    }
    }


(Tellvideo) #2

I was able to index the logs , i changed sending the indexing request directly to the datanode.


(system) #3