# Failed start of 2nd instance on same host with mlockall=true **URL:** \ **Category:** Elasticsearch\ **Created:** [August 26, 2014, 12:54pm UTC](https://discuss.elastic.co/t/failed-start-of-2nd-instance-on-same-host-with-mlockall-true/19463 "2014-08-26T12:54:50Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![R\_Toma](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@R\_Toma](https://discuss.elastic.co/u/R_Toma)\ **Post date:** [August 26, 2014, 12:54pm UTC](https://discuss.elastic.co/t/failed-start-of-2nd-instance-on-same-host-with-mlockall-true/19463/1 "2014-08-26T12:54:50Z")
Hi all, In an attempt to squeeze more power out of our physical servers we want to run multiple ES jvm's per server. Some specs: - servers has 24 cores, 256GB ram - each instance binds on different (alias) ip - each instance has 32GB heap - both instances run under user 'elastic' - limits for 'elastic' user: memlock=unlimited - es config for both instances: bootstrap.mlockall=true The 1st instance has been running for weeks. When starting the 2nd instance the following things happen: - increase of overal cpu load - lots of I/O to disks - no logging for 2nd instance - 2nd instance hangs - 1st instance keeps running, but gets slowish - cd /proc/ causes a hang of cd process (until 2nd instance is killed) - exec 'ps axuw' causes a hang of ps process (until 2nd instance is killed) Maybe (un)related: I have never been able to run Elasticsearch in a virtualbox with memlock=unlimited and mlockall=true. After an hour of trial & errors I found that removing setting 'bootstrap.mlockall' (setting it to false) from 2nd instance's configuration fixes things. I am confused, but acknowledge I do not know anything about memlocking. Any ideas? Regards, Renzo -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b5e4770a-4194-48c9-aec4-4919dc53342a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b5e4770a-4194-48c9-aec4-4919dc53342a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 26, 2014, 3:10pm UTC](https://discuss.elastic.co/t/failed-start-of-2nd-instance-on-same-host-with-mlockall-true/19463/2 "2014-08-26T15:10:48Z")
You should run one node per host. Two nodes add overhead and suffer from the effects you described. For mlockall, the user needs privilege to allocate the specified locked mem, and the OS need contiguous RAM per mlockall call. If the user's memlock limit is exhausted, or if RAM allocation gets fragmented, memlocking is no longer possible and fails. Jörg On Tue, Aug 26, 2014 at 2:54 PM, R. Toma [renzo.toma@gmail.com](mailto:renzo.toma@gmail.com) wrote: > Hi all, > > In an attempt to squeeze more power out of our physical servers we want to > run multiple ES jvm's per server. > > Some specs: > > - servers has 24 cores, 256GB ram > - each instance binds on different (alias) ip > - each instance has 32GB heap > - both instances run under user 'elastic' > - limits for 'elastic' user: memlock=unlimited > - es config for both instances: bootstrap.mlockall=true > > The 1st instance has been running for weeks. > > When starting the 2nd instance the following things happen: > > - increase of overal cpu load > - lots of I/O to disks > - no logging for 2nd instance > - 2nd instance hangs > - 1st instance keeps running, but gets slowish > - cd /proc/ causes a hang of cd process (until 2nd instance is killed) > - exec 'ps axuw' causes a hang of ps process (until 2nd instance is killed) > > Maybe (un)related: I have never been able to run Elasticsearch in a > virtualbox with memlock=unlimited and mlockall=true. > > After an hour of trial & errors I found that removing setting > 'bootstrap.mlockall' (setting it to false) from 2nd instance's > configuration fixes things. > > I am confused, but acknowledge I do not know anything about memlocking. > > Any ideas? > > Regards, > Renzo > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b5e4770a-4194-48c9-aec4-4919dc53342a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b5e4770a-4194-48c9-aec4-4919dc53342a%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/b5e4770a-4194-48c9-aec4-4919dc53342a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b5e4770a-4194-48c9-aec4-4919dc53342a%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGvtj3NKTWyMTjTre1FfJS31Khn%3DDAy\_kCxgVcCFpmDSQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGvtj3NKTWyMTjTre1FfJS31Khn%3DDAy_kCxgVcCFpmDSQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![R\_Toma](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@R\_Toma](https://discuss.elastic.co/u/R_Toma)\ **Post date:** [August 27, 2014, 7:53am UTC](https://discuss.elastic.co/t/failed-start-of-2nd-instance-on-same-host-with-mlockall-true/19463/3 "2014-08-27T07:53:35Z")
Found the following in the dmesg. Maybe I've hit a bug? INFO: task java:18056 blocked for more than 120 seconds. Not tainted 2.6.32-431.3.1.el6.x86\_64 #1 "echo 0 \> /proc/sys/kernel/hung\_task\_timeout\_secs" disables this message. java D 0000000000000002 0 18056 1 0x00000080 ffff883fe016fdc8 0000000000000082 0000000000000000 ffff883fe016fde8 ffff883fe016fd88 ffffffff8111f3f0 ffff881a89bc25d8 ffff883fe016fde8 ffff883edb4025f8 ffff883fe016ffd8 000000000000fbc8 ffff883edb4025f8 Call Trace: [] ? find\_get\_pages\_tag+0x40/0x130 [] ? prepare\_to\_wait+0x4e/0x80 [] jbd2\_log\_wait\_commit+0xc5/0x140 [jbd2] [] ? autoremove\_wake\_function+0x0/0x40 [] ? do\_writepages+0x21/0x40 [] jbd2\_complete\_transaction+0x68/0xb0 [jbd2] [] ext4\_sync\_file+0x121/0x1d0 [ext4] [] vfs\_fsync\_range+0xa1/0x100 [] vfs\_fsync+0x1d/0x20 [] do\_fsync+0x3e/0x60 [] sys\_fsync+0x10/0x20 [] system\_call\_fastpath+0x16/0x1b Op dinsdag 26 augustus 2014 14:54:50 UTC+2 schreef R. Toma: > Hi all, > > In an attempt to squeeze more power out of our physical servers we want to > run multiple ES jvm's per server. > > Some specs: > > - servers has 24 cores, 256GB ram > - each instance binds on different (alias) ip > - each instance has 32GB heap > - both instances run under user 'elastic' > - limits for 'elastic' user: memlock=unlimited > - es config for both instances: bootstrap.mlockall=true > > The 1st instance has been running for weeks. > > When starting the 2nd instance the following things happen: > > - increase of overal cpu load > - lots of I/O to disks > - no logging for 2nd instance > - 2nd instance hangs > - 1st instance keeps running, but gets slowish > - cd /proc/ causes a hang of cd process (until 2nd instance is killed) > - exec 'ps axuw' causes a hang of ps process (until 2nd instance is killed) > > Maybe (un)related: I have never been able to run Elasticsearch in a > virtualbox with memlock=unlimited and mlockall=true. > > After an hour of trial & errors I found that removing setting > 'bootstrap.mlockall' (setting it to false) from 2nd instance's > configuration fixes things. > > I am confused, but acknowledge I do not know anything about memlocking. > > Any ideas? > > Regards, > Renzo -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7fc4566e-631d-49f7-b012-3d1c2270102f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7fc4566e-631d-49f7-b012-3d1c2270102f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![R\_Toma](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@R\_Toma](https://discuss.elastic.co/u/R_Toma)\ **Post date:** [August 27, 2014, 7:59am UTC](https://discuss.elastic.co/t/failed-start-of-2nd-instance-on-same-host-with-mlockall-true/19463/4 "2014-08-27T07:59:49Z")
Hi Jörg, Running just 1 JVM with 32GB on a 24-core 256GB machine is a waste. CPU, I/O, memory metrics substantiate this. And off course we need to explore multi-instance before asking mgmt for more money. Regarding memlock: if no contiguous RAM is available I'd expect a fast error and not a totally hanging process and call traces (mentioning a 120 second timeouts) in the dmesg. Do you think this is maybe a jvm or elasticsearch bug? If so, i'll file it. Regards, Renzo Op dinsdag 26 augustus 2014 17:10:56 UTC+2 schreef Jörg Prante: > You should run one node per host. > > Two nodes add overhead and suffer from the effects you described. > > For mlockall, the user needs privilege to allocate the specified locked > mem, and the OS need contiguous RAM per mlockall call. If the user's > memlock limit is exhausted, or if RAM allocation gets fragmented, > memlocking is no longer possible and fails. > > Jörg > > On Tue, Aug 26, 2014 at 2:54 PM, R. Toma \<[renzo...@gmail.com](mailto:renzo...@gmail.com) \ > > > wrote: > > > Hi all, > > > > In an attempt to squeeze more power out of our physical servers we want > > to run multiple ES jvm's per server. > > > > Some specs: > > > > - servers has 24 cores, 256GB ram > > - each instance binds on different (alias) ip > > - each instance has 32GB heap > > - both instances run under user 'elastic' > > - limits for 'elastic' user: memlock=unlimited > > - es config for both instances: bootstrap.mlockall=true > > > > The 1st instance has been running for weeks. > > > > When starting the 2nd instance the following things happen: > > > > - increase of overal cpu load > > - lots of I/O to disks > > - no logging for 2nd instance > > - 2nd instance hangs > > - 1st instance keeps running, but gets slowish > > - cd /proc/ causes a hang of cd process (until 2nd instance is > > killed) > > - exec 'ps axuw' causes a hang of ps process (until 2nd instance is > > killed) > > > > Maybe (un)related: I have never been able to run Elasticsearch in a > > virtualbox with memlock=unlimited and mlockall=true. > > > > After an hour of trial & errors I found that removing setting > > 'bootstrap.mlockall' (setting it to false) from 2nd instance's > > configuration fixes things. > > > > I am confused, but acknowledge I do not know anything about memlocking. > > > > Any ideas? > > > > Regards, > > Renzo > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b5e4770a-4194-48c9-aec4-4919dc53342a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b5e4770a-4194-48c9-aec4-4919dc53342a%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/b5e4770a-4194-48c9-aec4-4919dc53342a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b5e4770a-4194-48c9-aec4-4919dc53342a%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dbe8fb95-2054-45ac-a07a-5bf2955e8869%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dbe8fb95-2054-45ac-a07a-5bf2955e8869%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:06am UTC](https://discuss.elastic.co/t/failed-start-of-2nd-instance-on-same-host-with-mlockall-true/19463/5 "2017-07-06T01:06:06Z")