Cluster setup

Elastic Stack Elasticsearch
1

Hi,

I have 3 ubuntu VM's on a private network, each has 64GB ram. I started ES
Beta2 (need it to use term vector feature) on each node with 30GB heap
space and with the following changes in the configuration file:

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["host1","host2","host3"]
bootstrap.mlockall: true
gateway.type: local
gateway.recover_after_nodes: 2
gateway.expected_nodes: 3
discovery.zen.minimum_master_nodes: 2

Sometimes es processes get killed by OOM killer and when I restart a node I
ran into situation like

host1,host2,host3 thinking all three nodes are connected and form a
cluster, and the host3 thinks it is all by itself with status red (names
are random)
host1 and host2 form one cluster and host2 and host3 forms another
cluster.

Any idea about what I am doing wrong and any suggestions? I will need to
index 20 million documents and have 3 separate indexes with 5 shards 1
replica.

Thanks,
T

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/62130b83-40aa-468e-9449-0046497d39ff%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hey,

I am not a huge fan of the OOM killer to be honest. However, something is
going against your plans when the OOM killer kicks in

You configured 30GB heap, but you are running out of memory (then most of
the time the process which takes the most memory is killed, obviously
elasticsearch). But why are you running out of memory? Do you have any
other service running on that machine, which eats up system memory? Please
check (or disable the OOM killer, but you should find out why it kicks in).

Also, use the nodes info API to find out if bootstrap.mlockall setting is
really configured correctly on your nodes.

--Alex

On Thu, Jan 16, 2014 at 5:50 PM, Tula tulay.muezzinoglu@gmail.com wrote:

Hi,

I have 3 ubuntu VM's on a private network, each has 64GB ram. I started
ES Beta2 (need it to use term vector feature) on each node with 30GB heap
space and with the following changes in the configuration file:

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["host1","host2","host3"]
bootstrap.mlockall: true
gateway.type: local
gateway.recover_after_nodes: 2
gateway.expected_nodes: 3
discovery.zen.minimum_master_nodes: 2

Sometimes es processes get killed by OOM killer and when I restart a node
I ran into situation like

host1,host2,host3 thinking all three nodes are connected and form a
cluster, and the host3 thinks it is all by itself with status red (names
are random)
host1 and host2 form one cluster and host2 and host3 forms another
cluster.

Any idea about what I am doing wrong and any suggestions? I will need to
index 20 million documents and have 3 separate indexes with 5 shards 1
replica.

Thanks,
T

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/62130b83-40aa-468e-9449-0046497d39ff%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_1C-pbsffn4pCkG5dNXKgJrvKEKF-SsBxoCR3cfRuA7A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Alex,

It looks like the issue was indeed not with the es itself but with our
virtual machine setup. We had to setup so that at least 2x heap space
reserved to our machines all the time. It has been running okay for a
month.

Thanks,
T

On Thursday, January 16, 2014 8:50:38 AM UTC-8, Tula wrote:

Hi,

I have 3 ubuntu VM's on a private network, each has 64GB ram. I started
ES Beta2 (need it to use term vector feature) on each node with 30GB heap
space and with the following changes in the configuration file:

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["host1","host2","host3"]
bootstrap.mlockall: true
gateway.type: local
gateway.recover_after_nodes: 2
gateway.expected_nodes: 3
discovery.zen.minimum_master_nodes: 2

Sometimes es processes get killed by OOM killer and when I restart a node
I ran into situation like

host1,host2,host3 thinking all three nodes are connected and form a
cluster, and the host3 thinks it is all by itself with status red (names
are random)
host1 and host2 form one cluster and host2 and host3 forms another
cluster.

Any idea about what I am doing wrong and any suggestions? I will need to
index 20 million documents and have 3 separate indexes with 5 shards 1
replica.

Thanks,
T

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1265e27a-6a02-40b8-b2d4-286b96c10429%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.