Starting multiple ES instances on one physical server

Elastic Stack Elasticsearch
1

Hi,

I am able to run multiple ES instances on one windows server from bat
file.
I want to run multiple ES instances through single or multiple ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please help me.

Thanks and Regards
Jagdeep

2

First, why do you need to run multiple ES instances? You don't really need
to... . Second, just run several instances of the service, or is that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh reach.jagdeep@gmail.comwrote:

Hi,

I am able to run multiple ES instances on one windows server from bat
file.
I want to run multiple ES instances through single or multiple ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please help me.

Thanks and Regards
Jagdeep

3

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than 1.5gb RAM
to one ES instance. I am doing term faceting for which I need more
memory.
Yes I tried to run several instances of service your your service
wrapper but unable to do that.

I want to control automatic restart if my ES instances if it goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You don't really need
to... . Second, just run several instances of the service, or is that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh reach.jagd...@gmail.comwrote:

Hi,

I am able to run multiple ES instances on one windows server from bat
file.
I want to run multiple ES instances through single or multiple ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please help me.

Thanks and Regards
Jagdeep

4

Clone your ES directory in 2 dir :
Es1
Es2

Then launch es1/bin/elastisearch.bat
And es2/bin/elastisearch.bat

IMHO it's a bad idea to run ES for production usage under Windows.

David :wink:
Twitter : @dadoonet / @elasticsearchfr

Le 2 mai 2012 à 18:39, jagdeep singh reach.jagdeep@gmail.com a écrit :

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than 1.5gb RAM
to one ES instance. I am doing term faceting for which I need more
memory.
Yes I tried to run several instances of service your your service
wrapper but unable to do that.
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

I want to control automatic restart if my ES instances if it goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You don't really need
to... . Second, just run several instances of the service, or is that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh reach.jagd...@gmail.comwrote:

Hi,

I am able to run multiple ES instances on one windows server from bat
file.
I want to run multiple ES instances through single or multiple ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please help me.

Thanks and Regards
Jagdeep

5

Are you saying you can only use the java windows 32bit? Is that because
thats the only one available in the servicewrapper community edition? Its a
shame if thats the case, you could try and use other way to start ES in
windows as a service, and use a 64bit JVM, it will be much better than
starting 10 instances of ES.... . Maybe try this:
Daemon – Daemon : Procrun.

On Wed, May 2, 2012 at 7:39 PM, jagdeep singh reach.jagdeep@gmail.comwrote:

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than 1.5gb RAM
to one ES instance. I am doing term faceting for which I need more
memory.
Yes I tried to run several instances of service your your service
wrapper but unable to do that.
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

I want to control automatic restart if my ES instances if it goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You don't really
need
to... . Second, just run several instances of the service, or is that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh <reach.jagd...@gmail.com
wrote:

Hi,

I am able to run multiple ES instances on one windows server from bat
file.
I want to run multiple ES instances through single or multiple ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please help me.

Thanks and Regards
Jagdeep

6

Ya David this is what I was doing earlier. But now I am starting
multiple instances from single location only.

You are right cant do this in production :frowning: I need proper service
configuration.

Thanks and Regards
Jagdeep

On May 2, 9:48 pm, David Pilato da...@pilato.fr wrote:

Clone your ES directory in 2 dir :
Es1
Es2

Then launch es1/bin/elastisearch.bat
And es2/bin/elastisearch.bat

IMHO it's a bad idea to run ES for production usage under Windows.

David :wink:
Twitter : @dadoonet / @elasticsearchfr

Le 2 mai 2012 à 18:39, jagdeep singh reach.jagd...@gmail.com a écrit :

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than 1.5gb RAM
to one ES instance. I am doing term faceting for which I need more
memory.
Yes I tried to run several instances of service your your service
wrapper but unable to do that.
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

I want to control automatic restart if my ES instances if it goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You don't really need
to... . Second, just run several instances of the service, or is that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh reach.jagd...@gmail.comwrote:

Hi,

I am able to run multiple ES instances on one windows server from bat
file.
I want to run multiple ES instances through single or multiple ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please help me.

Thanks and Regards
Jagdeep

7

You can not share the same data dir between your nodes.

David :wink:
Twitter : @dadoonet / @elasticsearchfr

Le 2 mai 2012 à 18:52, jagdeep singh reach.jagdeep@gmail.com a écrit :

Ya David this is what I was doing earlier. But now I am starting
multiple instances from single location only.

You are right cant do this in production :frowning: I need proper service
configuration.

Thanks and Regards
Jagdeep

On May 2, 9:48 pm, David Pilato da...@pilato.fr wrote:

Clone your ES directory in 2 dir :
Es1
Es2

Then launch es1/bin/elastisearch.bat
And es2/bin/elastisearch.bat

IMHO it's a bad idea to run ES for production usage under Windows.

David :wink:
Twitter : @dadoonet / @elasticsearchfr

Le 2 mai 2012 à 18:39, jagdeep singh reach.jagd...@gmail.com a écrit :

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than 1.5gb RAM
to one ES instance. I am doing term faceting for which I need more
memory.
Yes I tried to run several instances of service your your service
wrapper but unable to do that.
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

I want to control automatic restart if my ES instances if it goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You don't really need
to... . Second, just run several instances of the service, or is that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh reach.jagd...@gmail.comwrote:

Hi,

I am able to run multiple ES instances on one windows server from bat
file.
I want to run multiple ES instances through single or multiple ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please help me.

Thanks and Regards
Jagdeep

8

Yes "java windows 32bit" is a constraint as of now now. Using 64bit
service wrapper is not at all an issue. I am already using it on 64
bit OS but it has only 8 GB RAM.
I am trying to port to 64bit OS and jvm but it will take some time.
Will have to find an alternate by that time.
Unattended recovery of ES instances if it goes unresponsive is my main
concern for which I am not able to find appropriate approach.

Thanks and Regards
Jagdeep

On May 2, 9:49 pm, Shay Banon kim...@gmail.com wrote:

Are you saying you can only use the java windows 32bit? Is that because
thats the only one available in the servicewrapper community edition? Its a
shame if thats the case, you could try and use other way to start ES in
windows as a service, and use a 64bit JVM, it will be much better than
starting 10 instances of ES.... . Maybe try this:Daemon – Daemon : Procrun.

On Wed, May 2, 2012 at 7:39 PM, jagdeep singh reach.jagd...@gmail.comwrote:

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than 1.5gb RAM
to one ES instance. I am doing term faceting for which I need more
memory.
Yes I tried to run several instances of service your your service
wrapper but unable to do that.
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

I want to control automatic restart if my ES instances if it goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You don't really
need
to... . Second, just run several instances of the service, or is that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh <reach.jagd...@gmail.com
wrote:

Hi,

I am able to run multiple ES instances on one windows server from bat
file.
I want to run multiple ES instances through single or multiple ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please help me.

Thanks and Regards
Jagdeep

9

You mean that you get into problems when it runs OOM, and you can't recover
the data? Can you gist hte OOM failure you get in the logs in this case?

On Thu, May 3, 2012 at 8:14 AM, jagdeep singh reach.jagdeep@gmail.comwrote:

Yes "java windows 32bit" is a constraint as of now now. Using 64bit
service wrapper is not at all an issue. I am already using it on 64
bit OS but it has only 8 GB RAM.
I am trying to port to 64bit OS and jvm but it will take some time.
Will have to find an alternate by that time.
Unattended recovery of ES instances if it goes unresponsive is my main
concern for which I am not able to find appropriate approach.

Thanks and Regards
Jagdeep

On May 2, 9:49 pm, Shay Banon kim...@gmail.com wrote:

Are you saying you can only use the java windows 32bit? Is that because
thats the only one available in the servicewrapper community edition?
Its a
shame if thats the case, you could try and use other way to start ES in
windows as a service, and use a 64bit JVM, it will be much better than
starting 10 instances of ES.... . Maybe try this:
Daemon – Daemon : Procrun.

On Wed, May 2, 2012 at 7:39 PM, jagdeep singh <reach.jagd...@gmail.com
wrote:

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than 1.5gb RAM
to one ES instance. I am doing term faceting for which I need more
memory.
Yes I tried to run several instances of service your your service
wrapper but unable to do that.
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

I want to control automatic restart if my ES instances if it goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You don't really
need
to... . Second, just run several instances of the service, or is
that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Hi,

I am able to run multiple ES instances on one windows server from
bat
file.
I want to run multiple ES instances through single or multiple ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please help
me.

Thanks and Regards
Jagdeep

10

I am able to recover the data.. "there is no data loss" but I have to
manually restart the server after OOM.
Will post the gist also.

Thanks and Regards
Jagdeep

On May 4, 6:13 pm, Shay Banon kim...@gmail.com wrote:

You mean that you get into problems when it runs OOM, and you can't recover
the data? Can you gist hte OOM failure you get in the logs in this case?

On Thu, May 3, 2012 at 8:14 AM, jagdeep singh reach.jagd...@gmail.comwrote:

Yes "java windows 32bit" is a constraint as of now now. Using 64bit
service wrapper is not at all an issue. I am already using it on 64
bit OS but it has only 8 GB RAM.
I am trying to port to 64bit OS and jvm but it will take some time.
Will have to find an alternate by that time.
Unattended recovery of ES instances if it goes unresponsive is my main
concern for which I am not able to find appropriate approach.

Thanks and Regards
Jagdeep

On May 2, 9:49 pm, Shay Banon kim...@gmail.com wrote:

Are you saying you can only use the java windows 32bit? Is that because
thats the only one available in the servicewrapper community edition?
Its a
shame if thats the case, you could try and use other way to start ES in
windows as a service, and use a 64bit JVM, it will be much better than
starting 10 instances of ES.... . Maybe try this:
Daemon – Daemon : Procrun.

On Wed, May 2, 2012 at 7:39 PM, jagdeep singh <reach.jagd...@gmail.com
wrote:

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than 1.5gb RAM
to one ES instance. I am doing term faceting for which I need more
memory.
Yes I tried to run several instances of service your your service
wrapper but unable to do that.
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

I want to control automatic restart if my ES instances if it goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You don't really
need
to... . Second, just run several instances of the service, or is
that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Hi,

I am able to run multiple ES instances on one windows server from
bat
file.
I want to run multiple ES instances through single or multiple ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please help
me.

Thanks and Regards
Jagdeep

11

It might be needed, yea. Especially if the memory pressure does not relent
(for example, memory is being cleaned and then added immediately after).
You can configure the JVM to execute a script on OOM, potentially killing
the service.

On Fri, May 4, 2012 at 4:38 PM, jagdeep singh reach.jagdeep@gmail.comwrote:

I am able to recover the data.. "there is no data loss" but I have to
manually restart the server after OOM.
Will post the gist also.

Thanks and Regards
Jagdeep

On May 4, 6:13 pm, Shay Banon kim...@gmail.com wrote:

You mean that you get into problems when it runs OOM, and you can't
recover
the data? Can you gist hte OOM failure you get in the logs in this case?

On Thu, May 3, 2012 at 8:14 AM, jagdeep singh <reach.jagd...@gmail.com
wrote:

Yes "java windows 32bit" is a constraint as of now now. Using 64bit
service wrapper is not at all an issue. I am already using it on 64
bit OS but it has only 8 GB RAM.
I am trying to port to 64bit OS and jvm but it will take some time.
Will have to find an alternate by that time.
Unattended recovery of ES instances if it goes unresponsive is my main
concern for which I am not able to find appropriate approach.

Thanks and Regards
Jagdeep

On May 2, 9:49 pm, Shay Banon kim...@gmail.com wrote:

Are you saying you can only use the java windows 32bit? Is that
because
thats the only one available in the servicewrapper community edition?
Its a
shame if thats the case, you could try and use other way to start ES
in
windows as a service, and use a 64bit JVM, it will be much better
than
starting 10 instances of ES.... . Maybe try this:
Daemon – Daemon : Procrun.

On Wed, May 2, 2012 at 7:39 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than 1.5gb
RAM
to one ES instance. I am doing term faceting for which I need more
memory.
Yes I tried to run several instances of service your your service
wrapper but unable to do that.
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

I want to control automatic restart if my ES instances if it goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You don't
really
need
to... . Second, just run several instances of the service, or is
that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Hi,

I am able to run multiple ES instances on one windows server
from
bat
file.
I want to run multiple ES instances through single or multiple
ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please
help
me.

Thanks and Regards
Jagdeep

12

Hi Shay,

I have copied the log to gist

I am doing the term faceting and oom is expected and accepted for
that. But system should recover from this, it can't just go
unresponsive and need manual restart of the server. Please suggest if
I need to change number of nodes or configuration.

Thanks and Regards
Jagdeep

On May 4, 6:47 pm, Shay Banon kim...@gmail.com wrote:

It might be needed, yea. Especially if the memory pressure does not relent
(for example, memory is being cleaned and then added immediately after).
You can configure the JVM to execute a script on OOM, potentially killing
the service.

On Fri, May 4, 2012 at 4:38 PM, jagdeep singh reach.jagd...@gmail.comwrote:

I am able to recover the data.. "there is no data loss" but I have to
manually restart the server after OOM.
Will post the gist also.

Thanks and Regards
Jagdeep

On May 4, 6:13 pm, Shay Banon kim...@gmail.com wrote:

You mean that you get into problems when it runs OOM, and you can't
recover
the data? Can you gist hte OOM failure you get in the logs in this case?

On Thu, May 3, 2012 at 8:14 AM, jagdeep singh <reach.jagd...@gmail.com
wrote:

Yes "java windows 32bit" is a constraint as of now now. Using 64bit
service wrapper is not at all an issue. I am already using it on 64
bit OS but it has only 8 GB RAM.
I am trying to port to 64bit OS and jvm but it will take some time.
Will have to find an alternate by that time.
Unattended recovery of ES instances if it goes unresponsive is my main
concern for which I am not able to find appropriate approach.

Thanks and Regards
Jagdeep

On May 2, 9:49 pm, Shay Banon kim...@gmail.com wrote:

Are you saying you can only use the java windows 32bit? Is that
because
thats the only one available in the servicewrapper community edition?
Its a
shame if thats the case, you could try and use other way to start ES
in
windows as a service, and use a 64bit JVM, it will be much better
than
starting 10 instances of ES.... . Maybe try this:
Daemon – Daemon : Procrun.

On Wed, May 2, 2012 at 7:39 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than 1.5gb
RAM
to one ES instance. I am doing term faceting for which I need more
memory.
Yes I tried to run several instances of service your your service
wrapper but unable to do that.
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

I want to control automatic restart if my ES instances if it goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You don't
really
need
to... . Second, just run several instances of the service, or is
that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Hi,

I am able to run multiple ES instances on one windows server
from
bat
file.
I want to run multiple ES instances through single or multiple
ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please
help
me.

Thanks and Regards
Jagdeep

13

Well, OOM in Java is a strange beast, if you don't send any more requests
like the one you posted, which obviously can't be allocated, then the
system will probably recover. But if you continue to send more requests,
you will get into problems with the GC stalling the JVM.

Make sure you run a single node on the machine, and not several (I think
its the same one with the 32bit, which means you are not using the machine
resources properly), and allocate enough nodes to accommodate for it.

On Mon, May 7, 2012 at 6:08 PM, jagdeep singh reach.jagdeep@gmail.comwrote:

Hi Shay,

I have copied the log to gist
ES oom log and settings · GitHub

I am doing the term faceting and oom is expected and accepted for
that. But system should recover from this, it can't just go
unresponsive and need manual restart of the server. Please suggest if
I need to change number of nodes or configuration.

Thanks and Regards
Jagdeep

On May 4, 6:47 pm, Shay Banon kim...@gmail.com wrote:

It might be needed, yea. Especially if the memory pressure does not
relent
(for example, memory is being cleaned and then added immediately after).
You can configure the JVM to execute a script on OOM, potentially killing
the service.

On Fri, May 4, 2012 at 4:38 PM, jagdeep singh <reach.jagd...@gmail.com
wrote:

I am able to recover the data.. "there is no data loss" but I have to
manually restart the server after OOM.
Will post the gist also.

Thanks and Regards
Jagdeep

On May 4, 6:13 pm, Shay Banon kim...@gmail.com wrote:

You mean that you get into problems when it runs OOM, and you can't
recover
the data? Can you gist hte OOM failure you get in the logs in this
case?

On Thu, May 3, 2012 at 8:14 AM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Yes "java windows 32bit" is a constraint as of now now. Using 64bit
service wrapper is not at all an issue. I am already using it on 64
bit OS but it has only 8 GB RAM.
I am trying to port to 64bit OS and jvm but it will take some time.
Will have to find an alternate by that time.
Unattended recovery of ES instances if it goes unresponsive is my
main
concern for which I am not able to find appropriate approach.

Thanks and Regards
Jagdeep

On May 2, 9:49 pm, Shay Banon kim...@gmail.com wrote:

Are you saying you can only use the java windows 32bit? Is that
because
thats the only one available in the servicewrapper community
edition?
Its a
shame if thats the case, you could try and use other way to
start ES
in
windows as a service, and use a 64bit JVM, it will be much better
than
starting 10 instances of ES.... . Maybe try this:
Daemon – Daemon : Procrun.

On Wed, May 2, 2012 at 7:39 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than
1.5gb
RAM
to one ES instance. I am doing term faceting for which I need
more
memory.
Yes I tried to run several instances of service your your
service
wrapper but unable to do that.
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

I want to control automatic restart if my ES instances if it
goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You
don't
really
need
to... . Second, just run several instances of the service,
or is
that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Hi,

I am able to run multiple ES instances on one windows
server
from
bat
file.
I want to run multiple ES instances through single or
multiple
ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me. Please
help
me.

Thanks and Regards
Jagdeep

14

Thanks Shay for your response.

Ok then I am considering to upgrade my OS to 64 bit so that i can allocate
30 GB to 1 node only.

There is one more scenario on same server - Earlier we were setting nodes
as a combination of master and data, i.e. the
default configuration of nodes acting both as master and data. We moved out
of that configuration, and now each physical server is installed with 1
instance of master node and 1 or more instances of data node. Due to such
configuration, some data node might get unresponsive due to OOM, but the
master node doesn't suffer the problem, and continue providing responses by
gathering results from other data nodes as available.* In case of multiple *
*data node failure, results are provided with explicit OOM messages, which *
is even better. Using the timeout property in search queries didn't
respond
as expected, I don't know why, but provided it works, it will give the
perfect solution to our problem.

Can you please explain the detailed task division of master and data
nodes, and of the configuration of load-balancer node (neither master nor
data). We did not actually find much usage of it in accordance with our
setup.

Thanks and Regards
Jagdeep

On Wednesday, May 9, 2012 2:28:25 PM UTC+5:30, kimchy wrote:

Well, OOM in Java is a strange beast, if you don't send any more requests
like the one you posted, which obviously can't be allocated, then the
system will probably recover. But if you continue to send more requests,
you will get into problems with the GC stalling the JVM.

Make sure you run a single node on the machine, and not several (I think
its the same one with the 32bit, which means you are not using the machine
resources properly), and allocate enough nodes to accommodate for it.

On Mon, May 7, 2012 at 6:08 PM, jagdeep singh reach.jagdeep@gmail.comwrote:

Hi Shay,

I have copied the log to gist
ES oom log and settings · GitHub

I am doing the term faceting and oom is expected and accepted for
that. But system should recover from this, it can't just go
unresponsive and need manual restart of the server. Please suggest if
I need to change number of nodes or configuration.

Thanks and Regards
Jagdeep

On May 4, 6:47 pm, Shay Banon kim...@gmail.com wrote:

It might be needed, yea. Especially if the memory pressure does not
relent
(for example, memory is being cleaned and then added immediately after).
You can configure the JVM to execute a script on OOM, potentially
killing
the service.

On Fri, May 4, 2012 at 4:38 PM, jagdeep singh <reach.jagd...@gmail.com
wrote:

I am able to recover the data.. "there is no data loss" but I have to
manually restart the server after OOM.
Will post the gist also.

Thanks and Regards
Jagdeep

On May 4, 6:13 pm, Shay Banon kim...@gmail.com wrote:

You mean that you get into problems when it runs OOM, and you can't
recover
the data? Can you gist hte OOM failure you get in the logs in this
case?

On Thu, May 3, 2012 at 8:14 AM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Yes "java windows 32bit" is a constraint as of now now. Using
64bit
service wrapper is not at all an issue. I am already using it on
64
bit OS but it has only 8 GB RAM.
I am trying to port to 64bit OS and jvm but it will take some
time.
Will have to find an alternate by that time.
Unattended recovery of ES instances if it goes unresponsive is my
main
concern for which I am not able to find appropriate approach.

Thanks and Regards
Jagdeep

On May 2, 9:49 pm, Shay Banon kim...@gmail.com wrote:

Are you saying you can only use the java windows 32bit? Is that
because
thats the only one available in the servicewrapper community
edition?
Its a
shame if thats the case, you could try and use other way to
start ES
in
windows as a service, and use a 64bit JVM, it will be much
better
than
starting 10 instances of ES.... . Maybe try this:
Daemon – Daemon : Procrun.

On Wed, May 2, 2012 at 7:39 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than
1.5gb
RAM
to one ES instance. I am doing term faceting for which I need
more
memory.
Yes I tried to run several instances of service your your
service
wrapper but unable to do that.
GitHub - elastic/elasticsearch-servicewrapper: A service wrapper on top of elasticsearch

I want to control automatic restart if my ES instances if it
goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You
don't
really
need
to... . Second, just run several instances of the service,
or is
that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Hi,

I am able to run multiple ES instances on one windows
server
from
bat
file.
I want to run multiple ES instances through single or
multiple
ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me.
Please
help
me.

Thanks and Regards
Jagdeep

15

The master node job is simple, it is responsible to act to cluster wide
metadata or nodes changes. It will react to things like adding and removing
a node, or creating deleting an index, by maintaining a single view of the
global "cluster state".

When you say that you used single master node and multiple data nodes, did
you send requests to the master node?

The idea of a node balancer node (effectively, a node.client: true node) is
to simply be a node that is aware of the cluster state (more specifically,
shard placement). The best usage for it is for clients that use the HTTP
API connecting to "localhost" that runs the load balancer node, without
needing to worry about a list of addresses to connect to and handling
failover.

On Wed, May 9, 2012 at 12:36 PM, jagdeep singh reach.jagdeep@gmail.comwrote:

Thanks Shay for your response.

Ok then I am considering to upgrade my OS to 64 bit so that i can allocate
30 GB to 1 node only.

There is one more scenario on same server - Earlier we were setting
nodes as a combination of master and data, i.e. the
default configuration of nodes acting both as master and data. We moved
out
of that configuration, and now each physical server is installed with 1
instance of master node and 1 or more instances of data node. Due to such
configuration, some data node might get unresponsive due to OOM, but the
master node doesn't suffer the problem, and continue providing responses
by
gathering results from other data nodes as available.* In case of
multiple *
*data node failure, results are provided with explicit OOM messages,
which *
is even better. Using the timeout property in search queries didn't
respond
as expected, I don't know why, but provided it works, it will give the
perfect solution to our problem.

Can you please explain the detailed task division of master and data
nodes, and of the configuration of load-balancer node (neither master nor
data). We did not actually find much usage of it in accordance with our
setup.

Thanks and Regards
Jagdeep

On Wednesday, May 9, 2012 2:28:25 PM UTC+5:30, kimchy wrote:

Well, OOM in Java is a strange beast, if you don't send any more requests
like the one you posted, which obviously can't be allocated, then the
system will probably recover. But if you continue to send more requests,
you will get into problems with the GC stalling the JVM.

Make sure you run a single node on the machine, and not several (I think
its the same one with the 32bit, which means you are not using the machine
resources properly), and allocate enough nodes to accommodate for it.

On Mon, May 7, 2012 at 6:08 PM, jagdeep singh reach.jagdeep@gmail.comwrote:

Hi Shay,

I have copied the log to gist
https://gist.github.com/**2627823 https://gist.github.com/2627823

I am doing the term faceting and oom is expected and accepted for
that. But system should recover from this, it can't just go
unresponsive and need manual restart of the server. Please suggest if
I need to change number of nodes or configuration.

Thanks and Regards
Jagdeep

On May 4, 6:47 pm, Shay Banon kim...@gmail.com wrote:

It might be needed, yea. Especially if the memory pressure does not
relent
(for example, memory is being cleaned and then added immediately
after).
You can configure the JVM to execute a script on OOM, potentially
killing
the service.

On Fri, May 4, 2012 at 4:38 PM, jagdeep singh <reach.jagd...@gmail.com
**wrote:

I am able to recover the data.. "there is no data loss" but I have to
manually restart the server after OOM.
Will post the gist also.

Thanks and Regards
Jagdeep

On May 4, 6:13 pm, Shay Banon kim...@gmail.com wrote:

You mean that you get into problems when it runs OOM, and you can't
recover
the data? Can you gist hte OOM failure you get in the logs in this
case?

On Thu, May 3, 2012 at 8:14 AM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Yes "java windows 32bit" is a constraint as of now now. Using
64bit
service wrapper is not at all an issue. I am already using it on
64
bit OS but it has only 8 GB RAM.
I am trying to port to 64bit OS and jvm but it will take some
time.
Will have to find an alternate by that time.
Unattended recovery of ES instances if it goes unresponsive is
my main
concern for which I am not able to find appropriate approach.

Thanks and Regards
Jagdeep

On May 2, 9:49 pm, Shay Banon kim...@gmail.com wrote:

Are you saying you can only use the java windows 32bit? Is that
because
thats the only one available in the servicewrapper community
edition?
Its a
shame if thats the case, you could try and use other way to
start ES
in
windows as a service, and use a 64bit JVM, it will be much
better
than
starting 10 instances of ES.... . Maybe try this:
http://commons.apache.org/**daemon/procrun.htmlhttp://commons.apache.org/daemon/procrun.html
.

On Wed, May 2, 2012 at 7:39 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more than
1.5gb
RAM
to one ES instance. I am doing term faceting for which I
need more
memory.
Yes I tried to run several instances of service your your
service
wrapper but unable to do that.
https://github.com/**elasticsearch/elasticsearch-**
servicewrapperhttps://github.com/elasticsearch/elasticsearch-servicewrapper

I want to control automatic restart if my ES instances if it
goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You
don't
really
need
to... . Second, just run several instances of the service,
or is
that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Hi,

I am able to run multiple ES instances on one windows
server
from
bat
file.
I want to run multiple ES instances through single or
multiple
ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me.
Please
help
me.

Thanks and Regards
Jagdeep

16

Sorry for the late response. Missed ur reply :frowning:

1 master node and multiple data node is working fine as of now. Atleast I
am not entering into unresponsive state and can manage my nodes after
checking output from cluster health API. Shard allocation is bit slow in
this case. Since my master does not consume much memory and never ran oom,
I am wondering if multiple masters can further improve performance. Please
suggest.

This scenario can be useful for others too :slight_smile: atleast for ppl who have ram
allocation constraint.
Thanks for all your support.. looking forward for ur Berlinbuzzword talk :slight_smile:

Regards
Jagdeep

On Friday, May 11, 2012 2:48:33 AM UTC+5:30, kimchy wrote:

The master node job is simple, it is responsible to act to cluster wide
metadata or nodes changes. It will react to things like adding and removing
a node, or creating deleting an index, by maintaining a single view of the
global "cluster state".

When you say that you used single master node and multiple data nodes, did
you send requests to the master node?

The idea of a node balancer node (effectively, a node.client: true node)
is to simply be a node that is aware of the cluster state (more
specifically, shard placement). The best usage for it is for clients that
use the HTTP API connecting to "localhost" that runs the load balancer
node, without needing to worry about a list of addresses to connect to and
handling failover.

On Wed, May 9, 2012 at 12:36 PM, jagdeep singh reach.jagdeep@gmail.comwrote:

Thanks Shay for your response.

Ok then I am considering to upgrade my OS to 64 bit so that i can
allocate 30 GB to 1 node only.

There is one more scenario on same server - Earlier we were setting
nodes as a combination of master and data, i.e. the
default configuration of nodes acting both as master and data. We moved
out
of that configuration, and now each physical server is installed with 1
instance of master node and 1 or more instances of data node. Due to such
configuration, some data node might get unresponsive due to OOM, but the
master node doesn't suffer the problem, and continue providing responses
by
gathering results from other data nodes as available.* In case of
multiple *
*data node failure, results are provided with explicit OOM messages,
which *
is even better. Using the timeout property in search queries didn't
respond
as expected, I don't know why, but provided it works, it will give the
perfect solution to our problem.

Can you please explain the detailed task division of master and data
nodes, and of the configuration of load-balancer node (neither master nor
data). We did not actually find much usage of it in accordance with our
setup.

Thanks and Regards
Jagdeep

On Wednesday, May 9, 2012 2:28:25 PM UTC+5:30, kimchy wrote:

Well, OOM in Java is a strange beast, if you don't send any more
requests like the one you posted, which obviously can't be allocated, then
the system will probably recover. But if you continue to send more
requests, you will get into problems with the GC stalling the JVM.

Make sure you run a single node on the machine, and not several (I think
its the same one with the 32bit, which means you are not using the machine
resources properly), and allocate enough nodes to accommodate for it.

On Mon, May 7, 2012 at 6:08 PM, jagdeep singh reach.jagdeep@gmail.comwrote:

Hi Shay,

I have copied the log to gist
https://gist.github.com/**2627823 https://gist.github.com/2627823

I am doing the term faceting and oom is expected and accepted for
that. But system should recover from this, it can't just go
unresponsive and need manual restart of the server. Please suggest if
I need to change number of nodes or configuration.

Thanks and Regards
Jagdeep

On May 4, 6:47 pm, Shay Banon kim...@gmail.com wrote:

It might be needed, yea. Especially if the memory pressure does not
relent
(for example, memory is being cleaned and then added immediately
after).
You can configure the JVM to execute a script on OOM, potentially
killing
the service.

On Fri, May 4, 2012 at 4:38 PM, jagdeep singh <
reach.jagd...@gmail.com>**wrote:

I am able to recover the data.. "there is no data loss" but I have
to
manually restart the server after OOM.
Will post the gist also.

Thanks and Regards
Jagdeep

On May 4, 6:13 pm, Shay Banon kim...@gmail.com wrote:

You mean that you get into problems when it runs OOM, and you
can't
recover
the data? Can you gist hte OOM failure you get in the logs in
this case?

On Thu, May 3, 2012 at 8:14 AM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Yes "java windows 32bit" is a constraint as of now now. Using
64bit
service wrapper is not at all an issue. I am already using it
on 64
bit OS but it has only 8 GB RAM.
I am trying to port to 64bit OS and jvm but it will take some
time.
Will have to find an alternate by that time.
Unattended recovery of ES instances if it goes unresponsive is
my main
concern for which I am not able to find appropriate approach.

Thanks and Regards
Jagdeep

On May 2, 9:49 pm, Shay Banon kim...@gmail.com wrote:

Are you saying you can only use the java windows 32bit? Is
that
because
thats the only one available in the servicewrapper community
edition?
Its a
shame if thats the case, you could try and use other way to
start ES
in
windows as a service, and use a 64bit JVM, it will be much
better
than
starting 10 instances of ES.... . Maybe try this:
http://commons.apache.org/**daemon/procrun.htmlhttp://commons.apache.org/daemon/procrun.html
.

On Wed, May 2, 2012 at 7:39 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more
than 1.5gb
RAM
to one ES instance. I am doing term faceting for which I
need more
memory.
Yes I tried to run several instances of service your your
service
wrapper but unable to do that.
https://github.com/**elasticsearch/elasticsearch-**
servicewrapperhttps://github.com/elasticsearch/elasticsearch-servicewrapper

I want to control automatic restart if my ES instances if
it goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You
don't
really
need
to... . Second, just run several instances of the
service, or is
that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Hi,

I am able to run multiple ES instances on one windows
server
from
bat
file.
I want to run multiple ES instances through single or
multiple
ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me.
Please
help
me.

Thanks and Regards
Jagdeep

17

Heya, more "master eligible nodes" won't change things, since one effective
master node is elected and then used. Having a separate master node does
not really affect the perf of shard allocation..., the master node simply
decides that a shard should go to node X, and possibly recover its state
from a shard on node Y, then, the recovery happens from node X to node Y,
without the master being in the "middle".

On Mon, May 14, 2012 at 8:43 AM, jagdeep reach.jagdeep@gmail.com wrote:

Sorry for the late response. Missed ur reply :frowning:

1 master node and multiple data node is working fine as of now. Atleast I
am not entering into unresponsive state and can manage my nodes after
checking output from cluster health API. Shard allocation is bit slow in
this case. Since my master does not consume much memory and never ran oom,
I am wondering if multiple masters can further improve performance. Please
suggest.

This scenario can be useful for others too :slight_smile: atleast for ppl who have ram
allocation constraint.
Thanks for all your support.. looking forward for ur Berlinbuzzword talk :slight_smile:

Regards
Jagdeep

On Friday, May 11, 2012 2:48:33 AM UTC+5:30, kimchy wrote:

The master node job is simple, it is responsible to act to cluster wide
metadata or nodes changes. It will react to things like adding and removing
a node, or creating deleting an index, by maintaining a single view of the
global "cluster state".

When you say that you used single master node and multiple data nodes,
did you send requests to the master node?

The idea of a node balancer node (effectively, a node.client: true node)
is to simply be a node that is aware of the cluster state (more
specifically, shard placement). The best usage for it is for clients that
use the HTTP API connecting to "localhost" that runs the load balancer
node, without needing to worry about a list of addresses to connect to and
handling failover.

On Wed, May 9, 2012 at 12:36 PM, jagdeep singh reach.jagdeep@gmail.comwrote:

Thanks Shay for your response.

Ok then I am considering to upgrade my OS to 64 bit so that i can
allocate 30 GB to 1 node only.

There is one more scenario on same server - Earlier we were setting
nodes as a combination of master and data, i.e. the
default configuration of nodes acting both as master and data. We moved
out
of that configuration, and now each physical server is installed with 1
instance of master node and 1 or more instances of data node. Due to
such
configuration, some data node might get unresponsive due to OOM, but the
master node doesn't suffer the problem, and continue providing responses
by
gathering results from other data nodes as available.* In case of
multiple *
*data node failure, results are provided with explicit OOM messages,
which *
is even better. Using the timeout property in search queries didn't
respond
as expected, I don't know why, but provided it works, it will give the
perfect solution to our problem.

Can you please explain the detailed task division of master and data
nodes, and of the configuration of load-balancer node (neither master
nor
data). We did not actually find much usage of it in accordance with our
setup.

Thanks and Regards
Jagdeep

On Wednesday, May 9, 2012 2:28:25 PM UTC+5:30, kimchy wrote:

Well, OOM in Java is a strange beast, if you don't send any more
requests like the one you posted, which obviously can't be allocated, then
the system will probably recover. But if you continue to send more
requests, you will get into problems with the GC stalling the JVM.

Make sure you run a single node on the machine, and not several (I
think its the same one with the 32bit, which means you are not using the
machine resources properly), and allocate enough nodes to accommodate for
it.

On Mon, May 7, 2012 at 6:08 PM, jagdeep singh reach.jagdeep@gmail.comwrote:

Hi Shay,

I have copied the log to gist
https://gist.github.com/**262782**3 https://gist.github.com/2627823

I am doing the term faceting and oom is expected and accepted for
that. But system should recover from this, it can't just go
unresponsive and need manual restart of the server. Please suggest if
I need to change number of nodes or configuration.

Thanks and Regards
Jagdeep

On May 4, 6:47 pm, Shay Banon kim...@gmail.com wrote:

It might be needed, yea. Especially if the memory pressure does not
relent
(for example, memory is being cleaned and then added immediately
after).
You can configure the JVM to execute a script on OOM, potentially
killing
the service.

On Fri, May 4, 2012 at 4:38 PM, jagdeep singh <
reach.jagd...@gmail.com>wrote:

I am able to recover the data.. "there is no data loss" but I have
to
manually restart the server after OOM.
Will post the gist also.

Thanks and Regards
Jagdeep

On May 4, 6:13 pm, Shay Banon kim...@gmail.com wrote:

You mean that you get into problems when it runs OOM, and you
can't
recover
the data? Can you gist hte OOM failure you get in the logs in
this case?

On Thu, May 3, 2012 at 8:14 AM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Yes "java windows 32bit" is a constraint as of now now. Using
64bit
service wrapper is not at all an issue. I am already using it
on 64
bit OS but it has only 8 GB RAM.
I am trying to port to 64bit OS and jvm but it will take some
time.
Will have to find an alternate by that time.
Unattended recovery of ES instances if it goes unresponsive is
my main
concern for which I am not able to find appropriate approach.

Thanks and Regards
Jagdeep

On May 2, 9:49 pm, Shay Banon kim...@gmail.com wrote:

Are you saying you can only use the java windows 32bit? Is
that
because
thats the only one available in the servicewrapper community
edition?
Its a
shame if thats the case, you could try and use other way to
start ES
in
windows as a service, and use a 64bit JVM, it will be much
better
than
starting 10 instances of ES.... . Maybe try this:
http://commons.apache.org/**dae**mon/procrun.htmlhttp://commons.apache.org/daemon/procrun.html
.

On Wed, May 2, 2012 at 7:39 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Thanks Shay for your response.

I have 2 servers of following configuration
Windows 32 bit
24 core cpu
32 GB ram

Since its windows 32 bit I am not able to allocate more
than 1.5gb
RAM
to one ES instance. I am doing term faceting for which I
need more
memory.
Yes I tried to run several instances of service your your
service
wrapper but unable to do that.
https://github.com/**elasticsea**rch/elasticsearch-**
servicewrapp**erhttps://github.com/elasticsearch/elasticsearch-servicewrapper

I want to control automatic restart if my ES instances if
it goes
unresponsive. Any suggestions for this?

Thanks and rewgards
Jagdeep

On May 2, 9:28 pm, Shay Banon kim...@gmail.com wrote:

First, why do you need to run multiple ES instances? You
don't
really
need
to... . Second, just run several instances of the
service, or is
that a
problem?

On Tue, May 1, 2012 at 5:11 PM, jagdeep singh <
reach.jagd...@gmail.com
wrote:

Hi,

I am able to run multiple ES instances on one windows
server
from
bat
file.
I want to run multiple ES instances through single or
multiple
ES
services.
Server configuration:
Windows 32 bit
24 core cpu
32 GB ram

I am going to production so this is critical for me.
Please
help
me.

Thanks and Regards
Jagdeep