In an attempt to squeeze more power out of our physical servers we want to
run multiple ES jvm's per server.
Some specs:
servers has 24 cores, 256GB ram
each instance binds on different (alias) ip
each instance has 32GB heap
both instances run under user 'elastic'
limits for 'elastic' user: memlock=unlimited
es config for both instances: bootstrap.mlockall=true
The 1st instance has been running for weeks.
When starting the 2nd instance the following things happen:
increase of overal cpu load
lots of I/O to disks
no logging for 2nd instance
2nd instance hangs
1st instance keeps running, but gets slowish
cd /proc/ causes a hang of cd process (until 2nd instance is killed)
exec 'ps axuw' causes a hang of ps process (until 2nd instance is killed)
Maybe (un)related: I have never been able to run Elasticsearch in a
virtualbox with memlock=unlimited and mlockall=true.
After an hour of trial & errors I found that removing setting
'bootstrap.mlockall' (setting it to false) from 2nd instance's
configuration fixes things.
I am confused, but acknowledge I do not know anything about memlocking.
Two nodes add overhead and suffer from the effects you described.
For mlockall, the user needs privilege to allocate the specified locked
mem, and the OS need contiguous RAM per mlockall call. If the user's
memlock limit is exhausted, or if RAM allocation gets fragmented,
memlocking is no longer possible and fails.
In an attempt to squeeze more power out of our physical servers we want to
run multiple ES jvm's per server.
Some specs:
servers has 24 cores, 256GB ram
each instance binds on different (alias) ip
each instance has 32GB heap
both instances run under user 'elastic'
limits for 'elastic' user: memlock=unlimited
es config for both instances: bootstrap.mlockall=true
The 1st instance has been running for weeks.
When starting the 2nd instance the following things happen:
increase of overal cpu load
lots of I/O to disks
no logging for 2nd instance
2nd instance hangs
1st instance keeps running, but gets slowish
cd /proc/ causes a hang of cd process (until 2nd instance is killed)
exec 'ps axuw' causes a hang of ps process (until 2nd instance is killed)
Maybe (un)related: I have never been able to run Elasticsearch in a
virtualbox with memlock=unlimited and mlockall=true.
After an hour of trial & errors I found that removing setting
'bootstrap.mlockall' (setting it to false) from 2nd instance's
configuration fixes things.
I am confused, but acknowledge I do not know anything about memlocking.
Op dinsdag 26 augustus 2014 14:54:50 UTC+2 schreef R. Toma:
Hi all,
In an attempt to squeeze more power out of our physical servers we want to
run multiple ES jvm's per server.
Some specs:
servers has 24 cores, 256GB ram
each instance binds on different (alias) ip
each instance has 32GB heap
both instances run under user 'elastic'
limits for 'elastic' user: memlock=unlimited
es config for both instances: bootstrap.mlockall=true
The 1st instance has been running for weeks.
When starting the 2nd instance the following things happen:
increase of overal cpu load
lots of I/O to disks
no logging for 2nd instance
2nd instance hangs
1st instance keeps running, but gets slowish
cd /proc/ causes a hang of cd process (until 2nd instance is killed)
exec 'ps axuw' causes a hang of ps process (until 2nd instance is killed)
Maybe (un)related: I have never been able to run Elasticsearch in a
virtualbox with memlock=unlimited and mlockall=true.
After an hour of trial & errors I found that removing setting
'bootstrap.mlockall' (setting it to false) from 2nd instance's
configuration fixes things.
I am confused, but acknowledge I do not know anything about memlocking.
Running just 1 JVM with 32GB on a 24-core 256GB machine is a waste. CPU,
I/O, memory metrics substantiate this. And off course we need to explore
multi-instance before asking mgmt for more money.
Regarding memlock: if no contiguous RAM is available I'd expect a fast
error and not a totally hanging process and call traces (mentioning a 120
second timeouts) in the dmesg. Do you think this is maybe a jvm or
elasticsearch bug? If so, i'll file it.
Regards,
Renzo
Op dinsdag 26 augustus 2014 17:10:56 UTC+2 schreef Jörg Prante:
You should run one node per host.
Two nodes add overhead and suffer from the effects you described.
For mlockall, the user needs privilege to allocate the specified locked
mem, and the OS need contiguous RAM per mlockall call. If the user's
memlock limit is exhausted, or if RAM allocation gets fragmented,
memlocking is no longer possible and fails.
Jörg
On Tue, Aug 26, 2014 at 2:54 PM, R. Toma <renzo...@gmail.com <javascript:>
wrote:
Hi all,
In an attempt to squeeze more power out of our physical servers we want
to run multiple ES jvm's per server.
Some specs:
servers has 24 cores, 256GB ram
each instance binds on different (alias) ip
each instance has 32GB heap
both instances run under user 'elastic'
limits for 'elastic' user: memlock=unlimited
es config for both instances: bootstrap.mlockall=true
The 1st instance has been running for weeks.
When starting the 2nd instance the following things happen:
increase of overal cpu load
lots of I/O to disks
no logging for 2nd instance
2nd instance hangs
1st instance keeps running, but gets slowish
cd /proc/ causes a hang of cd process (until 2nd instance is
killed)
exec 'ps axuw' causes a hang of ps process (until 2nd instance is
killed)
Maybe (un)related: I have never been able to run Elasticsearch in a
virtualbox with memlock=unlimited and mlockall=true.
After an hour of trial & errors I found that removing setting
'bootstrap.mlockall' (setting it to false) from 2nd instance's
configuration fixes things.
I am confused, but acknowledge I do not know anything about memlocking.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.