Elasticsearch cluster Query

rrs · September 9, 2017, 10:37am

Hello,

We are in process of transferring logs over Elasticsearch will get 100GB data a day with 30 days retention policy. As per my understanding Elasticsearch layer requires more resources than Logstash & Kibana. We deciced provisioning two VM's with 8Core & 8GB RAM for both Logstash & Kibana and 3 VM's with same resources for Elasticsearch.

We don't have enough local storage to keep processed elasticserach data, hence we decided allocating nfs storage with 3TB. So, my question is 3TB volume will be mounted across all three elasticsearch nodes or 3*1 TB volumes get mounted on three nodes. Please suggest how my cluster will be equipped.

Thanks in advance.

warkolm · September 9, 2017, 9:32pm

We do not recommend using NFS as a direct store for Elasticsearch to run with. Even more so if you use it as a single shared volume, your performance will be horrible.

rrs · September 15, 2017, 7:26am

Hi Mark,

Thanks for quick help on this thread. For now, we going with SAN storage with 1TB on each node. As far as I know the beauty of Elasticsearch is it caters High Availability. As per that feature configured our Elasticsearch nodes as below that uses "split brain". Can you correct me, if I am wrong.

prd-els-1:
node.master: true
node.data: true
discovery.zen.ping.unicast.hosts: ["prd-els-1","prd-els-2","prd-els-3"]
discovery.zen.minimum_master_nodes: 2

prd-els-2:
node.master: true
node.data: true
discovery.zen.ping.unicast.hosts: ["prd-els-1","prd-els-2","prd-els-3"]
discovery.zen.minimum_master_nodes: 2

prd-els-3:
node.master: false
node.data: true
discovery.zen.ping.unicast.hosts: ["prd-els-1","prd-els-2","prd-els-3"]
discovery.zen.minimum_master_nodes: 2

Christian_Dahlqvist · September 15, 2017, 8:18am

Set node.master to true for the third node as well. This will allow a master to be elected even if you lose one node, and will make the cluster more resilient. Otherwise it looks good.

rrs · September 15, 2017, 11:55am

Hi,

Node of the configurations work to make Elasticserach up and running. I am seeing below errors, if we make 2 master + 1 non-master nodes and 3 master nodes as well. I also verified telnet is able to connect to both ports 9200/9300 from each host and its pair hosts.

The version we used elasticsearch-5.5.2-1.noarch installed on Centos-7.2. Can you please suggest.

[2017-09-15T11:47:19,491][INFO ][o.e.d.z.ZenDiscovery ] [prd-els-1] failed to send join request to master [{prd-els-3}{DQHiqOENQpWkKlb7WYAC6g}{KqE_1d7YQfaKpV2Et2kcVA}{prd-els-3}{10.xx.xxx.xxx:9300}], reason [RemoteTransportException[[prd-els-3][10.xx.xxx.xxx:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{prd-els-3}{DQHiqOENQpWkKlb7WYAC6g}{KqE_1d7YQfaKpV2Et2kcVA}{prd-els-3}{10.xx.xxx.xxx:9300}] not master for join request]; ], tried [3] times

Christian_Dahlqvist · September 15, 2017, 12:04pm

Is Elasticsearch starting up successfully on all nodes?

rrs · September 15, 2017, 12:10pm

Hi,

Yes, its getting started for every restart. The log output it generated is

[2017-09-15T12:06:45,624][INFO ][o.e.n.Node [2017-09-15T12:06:45,696][INFO ][o.e.e.NodeEnvironment [2017-09-15T12:06:45,696][INFO ][o.e.e.NodeEnvironment [2017-09-15T12:06:45,703][INFO ][o.e.n.Node [2017-09-15T12:06:45,704][INFO ][o.e.n.Node [2017-09-15T12:06:45,704][INFO ][o.e.n.Node [2017-09-15T12:06:46,563][INFO ][o.e.p.PluginsService [2017-09-15T12:06:46,563][INFO ][o.e.p.PluginsService [2017-09-15T12:06:46,563][INFO ][o.e.p.PluginsService [2017-09-15T12:06:46,563][INFO ][o.e.p.PluginsService [2017-09-15T12:06:46,563][INFO ][o.e.p.PluginsService [2017-09-15T12:06:46,563][INFO ][o.e.p.PluginsService [2017-09-15T12:06:46,563][INFO ][o.e.p.PluginsService [2017-09-15T12:06:46,563][INFO ][o.e.p.PluginsService [2017-09-15T12:06:46,564][INFO ][o.e.p.PluginsService [2017-09-15T12:06:46,564][INFO ][o.e.p.PluginsService [2017-09-15T12:06:46,564][INFO ][o.e.p.PluginsService [2017-09-15T12:06:46,564][INFO ][o.e.p.PluginsService [2017-09-15T12:06:47,941][INFO ][o.e.d.DiscoveryModule [2017-09-15T12:06:48,468][INFO ][o.e.n.Node [2017-09-15T12:06:48,468][INFO ][o.e.n.Node [2017-09-15T12:06:48,621][INFO ][o.e.t.TransportService [2017-09-15T12:06:48,633][INFO ][o.e.b.BootstrapChecks [2017-09-15T12:06:52,052][INFO ][o.e.d.z.ZenDiscovery ] [prd-els-1] initializing ...
] [prd-els-1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [9gb], net total_space [11.6gb], spins? [unknown], types [rootfs]
] [prd-els-1] heap size [2.9gb], compressed ordinary object pointers [true]
] [prd-els-1] node name [prd-els-1], node ID [DQHiqOENQpWkKlb7WYAC6g]
] [prd-els-1] version[5.5.2], pid[6594], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/3.10.0-327.18.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
] [prd-els-1] JVM arguments [-Xms3g, -Xmx3g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
] [prd-els-1] loaded module [aggs-matrix-stats]
] [prd-els-1] loaded module [ingest-common]
] [prd-els-1] loaded module [lang-expression]
] [prd-els-1] loaded module [lang-groovy]
] [prd-els-1] loaded module [lang-mustache]
] [prd-els-1] loaded module [lang-painless]
] [prd-els-1] loaded module [parent-join]
] [prd-els-1] loaded module [percolator]
] [prd-els-1] loaded module [reindex]
] [prd-els-1] loaded module [transport-netty3]
] [prd-els-1] loaded module [transport-netty4]
] [prd-els-1] no plugins loaded
] [prd-els-1] using discovery type [zen]
] [prd-els-1] initialized
] [prd-els-1] starting ...
] [prd-els-1] publish_address {10.xx.xxx.xxx:9300}, bound_addresses {10.xx.xxx.xxx:9300}
] [prd-els-1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
] [prd-els-1] failed to send join request to master [{prd-els-3}{DQHiqOENQpWkKlb7WYAC6g}{KqE_1d7YQfaKpV2Et2kcVA}{prd-els-3}{10.xx.xxx.xxx:9300}], reason [RemoteTransportException[[prd-els-3][10.xx.xxx.xxx:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{prd-els-3}{DQHiqOENQpWkKlb7WYAC6g}{KqE_1d7YQfaKpV2Et2kcVA}{prd-els-3}{10.xx.xxx.xxx:9300}] not master for join request]; ], tried [3] times

rrs · September 15, 2017, 12:17pm

Hi,

And, the configuration we made is as below and ensured elasticserach has proper privileges to write logs/indexes.

One of the master node:
cluster.name: prod_app_01
node.name: prd-els-2
node.master: true
node.data: true
path.data: /lsh/data
path.logs: /lsh/logs
network.host: 10.xx.xxx.102
http.port: 9200
discovery.zen.ping.unicast.hosts: ["prd-els-1", "prd-els-2", "prd-els-3"]
discovery.zen.minimum_master_nodes: 2
network.publish_host: prd-els-2

One of the data_node:
cluster.name: prod_app_01
node.name: prd-els-3
node.master: true
node.data: true
path.data: /lsh/data
path.logs: /lsh/logs
network.host: 10.xx.xxx.103
http.port: 9200
discovery.zen.ping.unicast.hosts: ["prd-els-1", "prd-els-2", "prd-els-3"]
discovery.zen.minimum_master_nodes: 2
network.publish_host: prd-els-3

rrs · September 15, 2017, 2:54pm

Hi Christian_Dahlqvist / Mark,

Thanks for your support. I have fixed the issue now cluster status looks pretty good. The RCA is that since my VMWare supports cloning, first I installed Elasticserach one one VM with required parameters then I cloned same VM to another two nodes.

As I know Elasticsearch uses ID's the same ID is replicated for other two nodes while cloning, which causes Elasticsearch to see only one ID even though there are 3 nodes. I realized and re-installed/re-configured elasticserach. Now, it works as expected.

~]# curl -s 'prd-els-1:9200/_cat/nodes'
10.120.181.129 11 85 6 0.34 0.13 0.08 mdi - prd-els-1
10.120.181.131 16 53 3 0.06 0.06 0.05 mdi * prd-els-2
10.120.181.132 11 78 2 0.05 0.07 0.06 di - prd-els-3

One last question is we have Logstash layer installed on another VM where logstash forwards processed logs to prd-els-1 which is one of my master that distributes indexes across other nodes.

If my prd-els-1 goes down, how Logstash sends the new logs?
And, my kibana config points to primary master prd-els-1. How can I search logs?

Please advice.

Christian_Dahlqvist · September 15, 2017, 2:59pm

You should always aim to have 3 master-eligible nodes and set minimum_master_nodes to 2. Logstash should have all 3 nodes defined so that it can distribute load across the cluster.

rrs · September 15, 2017, 3:12pm

Hi,

It makes sense. We follow it to make all 3 nodes as master-eligible. Due to resources crunch and this is initial setup the cluster will act as making indexes and serving seach traffic as well. Could you also suggest best how it can be fulfilled with existing 3 node cluster and below points

If my prd-els-1 goes down, how Logstash sends the new logs?
And, my kibana config points to primary master prd-els-1. How can I search logs?

system · October 13, 2017, 3:12pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.