ES freezes the server at startup


(Godefroy-2) #1

Hello,

I discovered ElasticSearch few weeks ago and I use it for a new
project. It's a great tool !
On my laptop (Ubuntu), I have no problem with ES.

But when I try to start ES on a server (Debian), the server totally
freezes and I have to hard-reboot it.
I use the ES out of the box :
http://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.13.1.tar.gz
without configuration for the moment.
And I start it with bin/elasticsearch -f
(I also tried with the service package, same problem)

I tried on two servers :

  • 1 Dedibox (Online.net), now the server is dead...
  • 1 Kimsufi (Kimsufi.com), I fortunately can reboot it without problem

Do you have some idea to fix this issue or to correctly configure ES ?

Thanks,
Godefroy


(Clinton Gormley) #2

Hiya

But when I try to start ES on a server (Debian), the server totally
freezes and I have to hard-reboot it.

What version of Java are you using? If it's not the official Sun Java,
you may run into problems (based on my experience with other Java
software, like Eclipse)

clint


(Shay Banon) #3

Strange, that its completely crashing the os... . One thing that is done
natively now in 0.13 is using mlockall, might be a good try to disable it
and see if it helps:
http://www.elasticsearch.com/docs/elasticsearch/setup/installation/#Important_Configurations
.

On Fri, Dec 17, 2010 at 9:13 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

Hiya

But when I try to start ES on a server (Debian), the server totally
freezes and I have to hard-reboot it.

What version of Java are you using? If it's not the official Sun Java,
you may run into problems (based on my experience with other Java
software, like Eclipse)

clint


(Godefroy-2) #4

Thanks for your answers!

I use openjdk-6-jre
I just tried with sun-java6-jre and I've the same pb.

With bootstrap.mlockall set to false, ES starts !
I have this exception, but ES is usable :
Exception in thread "elasticsearch[Jubilee][tp]-pool-1-thread-1"
org.elasticsearch.discovery.zen.ping.ZenPingException: Failed to send
ping request over multicast

Why do ES crashes when bootstrap.mlockall is set to true ? It seems to
be dangerous... (It costs me 1 server, and I've lost lot of data)

On 17 déc, 20:16, Shay Banon shay.ba...@elasticsearch.com wrote:

Strange, that its completely crashing the os... . One thing that is done
natively now in 0.13 is using mlockall, might be a good try to disable it
and see if it helps:http://www.elasticsearch.com/docs/elasticsearch/setup/installation/#I...
.

On Fri, Dec 17, 2010 at 9:13 PM, Clinton Gormley clin...@iannounce.co.ukwrote:

Hiya

But when I try to start ES on a server (Debian), the server totally
freezes and I have to hard-reboot it.

What version of Java are you using? If it's not the official Sun Java,
you may run into problems (based on my experience with other Java
software, like Eclipse)

clint


(Shay Banon) #5

I am not sure why it crashes to be honest, it just calls mlockall, it
shouldn't cause the OS to crash completely. In any case, let me turn this to
be off by default, and allow people to turn it on.

-shay.banon

On Fri, Dec 17, 2010 at 11:38 PM, Godefroy compreignac@gmail.com wrote:

Thanks for your answers!

I use openjdk-6-jre
I just tried with sun-java6-jre and I've the same pb.

With bootstrap.mlockall set to false, ES starts !
I have this exception, but ES is usable :
Exception in thread "elasticsearch[Jubilee][tp]-pool-1-thread-1"
org.elasticsearch.discovery.zen.ping.ZenPingException: Failed to send
ping request over multicast

Why do ES crashes when bootstrap.mlockall is set to true ? It seems to
be dangerous... (It costs me 1 server, and I've lost lot of data)

On 17 déc, 20:16, Shay Banon shay.ba...@elasticsearch.com wrote:

Strange, that its completely crashing the os... . One thing that is done
natively now in 0.13 is using mlockall, might be a good try to disable it
and see if it helps:
http://www.elasticsearch.com/docs/elasticsearch/setup/installation/#I...
.

On Fri, Dec 17, 2010 at 9:13 PM, Clinton Gormley <
clin...@iannounce.co.uk>wrote:

Hiya

But when I try to start ES on a server (Debian), the server totally
freezes and I have to hard-reboot it.

What version of Java are you using? If it's not the official Sun Java,
you may run into problems (based on my experience with other Java
software, like Eclipse)

clint


(Clinton Gormley) #6

Why do ES crashes when bootstrap.mlockall is set to true ? It seems to
be dangerous... (It costs me 1 server, and I've lost lot of data)

What are your min and max memory settings, and how much memory do you
have available on the crashing server?

Could it be that you're trying to assign more memory than you have,
which causes all of your other processes to swap out?

clint


(Godefroy-2) #7

Yes I think that I've not enough memory.
But ES should check if there is enough memory before the mlockall,
nope ?

On 19 déc, 12:49, Clinton Gormley clin...@iannounce.co.uk wrote:

Why do ES crashes when bootstrap.mlockall is set to true ? It seems to
be dangerous... (It costs me 1 server, and I've lost lot of data)

What are your min and max memory settings, and how much memory do you
have available on the crashing server?

Could it be that you're trying to assign more memory than you have,
which causes all of your other processes to swap out?

clint


(ppearcy) #8

My two cents... This feature makes ES performance very reliable and
fixed some performance spikes that we encountered. I think it should
be defaulted to on.

It seems to me that the setting only causes problems when misused.
Along these same lines, what happens without this setting and someone
tries to allocate more mem than their system has and their JVM heap
reaches that point? I'd assume massive swapping leading to performance
degradation, which would only happen after hours or days of up time.
This seems like a ticking time bomb that would be much worse than fail
to start up.

Best case would be to check system memory before the mlockall call and
then gracefully exit with a descriptive error that you don't have
enough mem on your system.

Btw, this should probably be listed as a breaking change in the 0.14
release notes.

Thanks,
Paul

On Dec 27, 4:30 pm, Godefroy compreig...@gmail.com wrote:

Yes I think that I've not enough memory.
But ES should check if there is enough memory before themlockall,
nope ?

On 19 déc, 12:49, Clinton Gormley clin...@iannounce.co.uk wrote:

Why do ES crashes when bootstrap.mlockallis set to true ? It seems to
be dangerous... (It costs me 1 server, and I've lost lot of data)

What are your min and max memory settings, and how much memory do you
have available on the crashing server?

Could it be that you're trying to assign more memory than you have,
which causes all of your other processes to swap out?

clint


(Shay Banon) #9

The problem with having this setting set to true is the fact that people
have reported that it got to a stage where the OS crashed completely. Not
sure why it happened, but ES should not cause that with an out of the box
setting. Checking if there is enough memory is a good start, there is no API
for that in Java, but can try and use native code, I can play around with it
and see how it goes.

On Tue, Dec 28, 2010 at 7:14 PM, Paul ppearcy@gmail.com wrote:

My two cents... This feature makes ES performance very reliable and
fixed some performance spikes that we encountered. I think it should
be defaulted to on.

It seems to me that the setting only causes problems when misused.
Along these same lines, what happens without this setting and someone
tries to allocate more mem than their system has and their JVM heap
reaches that point? I'd assume massive swapping leading to performance
degradation, which would only happen after hours or days of up time.
This seems like a ticking time bomb that would be much worse than fail
to start up.

Best case would be to check system memory before the mlockall call and
then gracefully exit with a descriptive error that you don't have
enough mem on your system.

Btw, this should probably be listed as a breaking change in the 0.14
release notes.

Thanks,
Paul

On Dec 27, 4:30 pm, Godefroy compreig...@gmail.com wrote:

Yes I think that I've not enough memory.
But ES should check if there is enough memory before themlockall,
nope ?

On 19 déc, 12:49, Clinton Gormley clin...@iannounce.co.uk wrote:

Why do ES crashes when bootstrap.mlockallis set to true ? It seems to
be dangerous... (It costs me 1 server, and I've lost lot of data)

What are your min and max memory settings, and how much memory do you
have available on the crashing server?

Could it be that you're trying to assign more memory than you have,
which causes all of your other processes to swap out?

clint


(ppearcy) #10

Sigar can provide you this value in Java space with a call to
sigar.getMem().getTotal().

Thanks

On Dec 28, 11:15 am, Shay Banon shay.ba...@elasticsearch.com wrote:

The problem with having this setting set to true is the fact that people
have reported that it got to a stage where the OS crashed completely. Not
sure why it happened, but ES should not cause that with an out of the box
setting. Checking if there is enough memory is a good start, there is no API
for that in Java, but can try and use native code, I can play around with it
and see how it goes.

On Tue, Dec 28, 2010 at 7:14 PM, Paul ppea...@gmail.com wrote:

My two cents... This feature makes ES performance very reliable and
fixed some performance spikes that we encountered. I think it should
be defaulted to on.

It seems to me that the setting only causes problems when misused.
Along these same lines, what happens without this setting and someone
tries to allocate more mem than their system has and their JVM heap
reaches that point? I'd assume massive swapping leading to performance
degradation, which would only happen after hours or days of up time.
This seems like a ticking time bomb that would be much worse than fail
to start up.

Best case would be to check system memory before the mlockall call and
then gracefully exit with a descriptive error that you don't have
enough mem on your system.

Btw, this should probably be listed as a breaking change in the 0.14
release notes.

Thanks,
Paul

On Dec 27, 4:30 pm, Godefroy compreig...@gmail.com wrote:

Yes I think that I've not enough memory.
But ES should check if there is enough memory before themlockall,
nope ?

On 19 déc, 12:49, Clinton Gormley clin...@iannounce.co.uk wrote:

Why do ES crashes when bootstrap.mlockallis set to true ? It seems to
be dangerous... (It costs me 1 server, and I've lost lot of data)

What are your min and max memory settings, and how much memory do you
have available on the crashing server?

Could it be that you're trying to assign more memory than you have,
which causes all of your other processes to swap out?

clint


(Shay Banon) #11

Agreed, thats what I meant when I said going native (ES already uses Sigar)
:). Though, total mem might not be enough, but available mem should be
checked.

On Tue, Dec 28, 2010 at 8:23 PM, Paul ppearcy@gmail.com wrote:

Sigar can provide you this value in Java space with a call to
sigar.getMem().getTotal().

Thanks

On Dec 28, 11:15 am, Shay Banon shay.ba...@elasticsearch.com wrote:

The problem with having this setting set to true is the fact that people
have reported that it got to a stage where the OS crashed completely. Not
sure why it happened, but ES should not cause that with an out of the box
setting. Checking if there is enough memory is a good start, there is no
API
for that in Java, but can try and use native code, I can play around with
it
and see how it goes.

On Tue, Dec 28, 2010 at 7:14 PM, Paul ppea...@gmail.com wrote:

My two cents... This feature makes ES performance very reliable and
fixed some performance spikes that we encountered. I think it should
be defaulted to on.

It seems to me that the setting only causes problems when misused.
Along these same lines, what happens without this setting and someone
tries to allocate more mem than their system has and their JVM heap
reaches that point? I'd assume massive swapping leading to performance
degradation, which would only happen after hours or days of up time.
This seems like a ticking time bomb that would be much worse than fail
to start up.

Best case would be to check system memory before the mlockall call and
then gracefully exit with a descriptive error that you don't have
enough mem on your system.

Btw, this should probably be listed as a breaking change in the 0.14
release notes.

Thanks,
Paul

On Dec 27, 4:30 pm, Godefroy compreig...@gmail.com wrote:

Yes I think that I've not enough memory.
But ES should check if there is enough memory before themlockall,
nope ?

On 19 déc, 12:49, Clinton Gormley clin...@iannounce.co.uk wrote:

Why do ES crashes when bootstrap.mlockallis set to true ? It
seems to

be dangerous... (It costs me 1 server, and I've lost lot of data)

What are your min and max memory settings, and how much memory do
you

have available on the crashing server?

Could it be that you're trying to assign more memory than you have,
which causes all of your other processes to swap out?

clint


(system) #12