Hi Magnus,
Need your help on the following issue i am facing
I have done a elastic search clustering with two nodes.
Both of them are master nodes as well as data nodes.
Data to the nodes passes through two different instances of logstash.
Lets say ES1 and ES2 are two elastic search nodes.
LS1 and LS2 are two logstash instances respectively servingfor node ES1 and ES2
Kibana is deployed and points to ES1
Now data goes to two different nodes of ES1 and ES2 by LS1 and LS2 respectively.
Kibana connects to ES1 which is deployed in same virtual machines and should be able to read data from both ES1 and ES2.
But the data displayed in kibana just belongs to ES1.
Below is the configuration for the elasticsearch.yml file for both the instances.
instance -1
####################################
cluster.name: fvanalytics
node.name: "R4PVAP1030-MON"
node.master: true
node.data: true
threadpool.search.queue_size: 10000
indices.fielddata.cache.size: 70%
indices.breaker.fielddata.limit: 85%
transport.tcp.port: 9300
http.port: 9200
#######################
instance -2
####################################
cluster.name: fvanalytics
node.name: "R4PVAP1031-MON"
node.master: true
node.data: true
threadpool.search.queue_size: 10000
indices.fielddata.cache.size: 70%
indices.breaker.fielddata.limit: 85%
transport.tcp.port: 9300
http.port: 9200
#######################
Firewall port 9300 is opened for both instances of ES in both the VMs
Please correct me where i have gone wrong.
regards
Dibya
The nodes need to be given a unicast list so they know where to find other nodes. Add this setting to the config of both nodes:
discovery.zen.ping.unicast.hosts: ["host1", "host2"]
Also, unless you have a very good reason, these configurations are very dangerous and just asking for trouble. I'd remove them:
threadpool.search.queue_size: 10000
indices.fielddata.cache.size: 70%
Hi ,
Thanks for a quick reply
cache i had to increase considering the data size and limited RAM.
Why is threadpool size dangerous?
host1 and host2 should be IPAddress or hostname simply?
regards
Dibyananda
Either hostname or IP should work, as long as your internal network will resolve the hostname. I tend to use IP personally.
I would still be careful with the cache size. Much better to use doc values and ignore field data cache entirely.
A large threadpool queue is an anti-pattern because it hides problems. The threadpool queue exists to absorb transient bursts of activity. If you require a very large queue, that's a good indication that your cluster is under-sized for the amount of activity. A large queue may work now, but it's just a temporary solution: at some point you'll likely need to scale out.
Some threadpools are more dangerous than others to increase. For example, a large bulk queue is dangerous because the queue is in-memory only and not committed to disk (because it's queued for processing), so if the node goes down all the bulk actions are lost and you may "lose" data because it was never inserted into ES.
Full queues are supposed to act as back-pressure to force your application to slow down, since the cluster is operating at-capacity. 
1 Like