# Engine failure, message \[OutOfMemoryError\[unable to create new native thread\]\] **URL:** \ **Category:** Elasticsearch\ **Created:** [February 11, 2014, 5:45am UTC](https://discuss.elastic.co/t/engine-failure-message-outofmemoryerror-unable-to-create-new-native-thread/15717 "2014-02-11T05:45:01Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![T\_Vinod\_Gupta](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@T\_Vinod\_Gupta](https://discuss.elastic.co/u/T_Vinod_Gupta)\ **Post date:** [February 11, 2014, 5:45am UTC](https://discuss.elastic.co/t/engine-failure-message-outofmemoryerror-unable-to-create-new-native-thread/15717/1 "2014-02-11T05:45:01Z")
hi, i had a stable ES cluster on aws ec2 instances till a week ago.. and i don't know whats going on and my cluster keeps getting into a bad state every few hours. the error says OOM but i know that that is not the reason. the instance has enough heap space left. im running ES 0.90.6 version and giving half the ram (8gb) to ES process. and i see these messages (the same message kind of) in the logs on all the machines in the cluster. [2014-02-11 03:17:39,936][WARN][cluster.action.shard] [Star-Dancer] [facebook\_022014][1] sending failed shard for [facebook\_022014][1], node[zO9Pc1GNSuiVMA\_Kn2b3UQ], [R], s[STARTED], indexUUID [qN3CUSfVS-m2KlgQQtOqxg], reason [engine failure, message [OutOfMemoryError[unable to create new native thread]]] any ideas on how to debug this or how to figure out whats causing this would be really helpful. thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHau4ytfEHERAsVcGFc-R8yePmw\_KaN%3DJDV-cEwm9VK2VLgp-Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHau4ytfEHERAsVcGFc-R8yePmw_KaN%3DJDV-cEwm9VK2VLgp-Q%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 11, 2014, 8:10am UTC](https://discuss.elastic.co/t/engine-failure-message-outofmemoryerror-unable-to-create-new-native-thread/15717/2 "2014-02-11T08:10:54Z")
Your user ran out of thread/process space. This is reported as OOM in Java. You can check the "nproc" entry in /etc/security.d/limits.conf for maximum settings and compare this with the process table. The OS settings regarding threads are usually ok and should not be modified. Check if you have modified ES default settings regarding the thread pools, and revert this changes to the default settings. If this does not help, you should upgrade from 0.90.6 to 0.90.11 Jörg On Tue, Feb 11, 2014 at 6:45 AM, T Vinod Gupta [tvinod@readypulse.com](mailto:tvinod@readypulse.com)wrote: > hi, > i had a stable ES cluster on aws ec2 instances till a week ago.. and i > don't know whats going on and my cluster keeps getting into a bad state > every few hours. the error says OOM but i know that that is not the reason. > the instance has enough heap space left. im running ES 0.90.6 version and > giving half the ram (8gb) to ES process. and i see these messages (the same > message kind of) in the logs on all the machines in the cluster. > > [2014-02-11 03:17:39,936][WARN][cluster.action.shard] [Star-Dancer] > [facebook\_022014][1] sending failed shard for [facebook\_022014][1], > node[zO9Pc1GNSuiVMA\_Kn2b3UQ], [R], s[STARTED], indexUUID > [qN3CUSfVS-m2KlgQQtOqxg], reason [engine failure, message > [OutOfMemoryError[unable to create new native thread]]] > > any ideas on how to debug this or how to figure out whats causing this > would be really helpful. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGnDrE-yZnyFgUbMks84KVsyB%3Dp\_9UGvQ\_DmUo5Diub0g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGnDrE-yZnyFgUbMks84KVsyB%3Dp_9UGvQ_DmUo5Diub0g%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![T\_Vinod\_Gupta](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@T\_Vinod\_Gupta](https://discuss.elastic.co/u/T_Vinod_Gupta)\ **Post date:** [February 25, 2014, 9:19am UTC](https://discuss.elastic.co/t/engine-failure-message-outofmemoryerror-unable-to-create-new-native-thread/15717/3 "2014-02-25T09:19:07Z")
thanks for your response Jörg, somehow missed replying earlier. for some strange reason, the max threads setting was reset when i did a reboot.. so i had to set it back to a high number. On Tue, Feb 11, 2014 at 12:10 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > Your user ran out of thread/process space. This is reported as OOM in > Java. > > You can check the "nproc" entry in /etc/security.d/limits.conf for maximum > settings and compare this with the process table. > > The OS settings regarding threads are usually ok and should not be > modified. Check if you have modified ES default settings regarding the > thread pools, and revert this changes to the default settings. If this does > not help, you should upgrade from 0.90.6 to 0.90.11 > > Jörg > > On Tue, Feb 11, 2014 at 6:45 AM, T Vinod Gupta [tvinod@readypulse.com](mailto:tvinod@readypulse.com)wrote: > > > hi, > > i had a stable ES cluster on aws ec2 instances till a week ago.. and i > > don't know whats going on and my cluster keeps getting into a bad state > > every few hours. the error says OOM but i know that that is not the reason. > > the instance has enough heap space left. im running ES 0.90.6 version and > > giving half the ram (8gb) to ES process. and i see these messages (the same > > message kind of) in the logs on all the machines in the cluster. > > > > [2014-02-11 03:17:39,936][WARN][cluster.action.shard] [Star-Dancer] > > [facebook\_022014][1] sending failed shard for [facebook\_022014][1], > > node[zO9Pc1GNSuiVMA\_Kn2b3UQ], [R], s[STARTED], indexUUID > > [qN3CUSfVS-m2KlgQQtOqxg], reason [engine failure, message > > [OutOfMemoryError[unable to create new native thread]]] > > > > any ideas on how to debug this or how to figure out whats causing this > > would be really helpful. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGnDrE-yZnyFgUbMks84KVsyB%3Dp\_9UGvQ\_DmUo5Diub0g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGnDrE-yZnyFgUbMks84KVsyB%3Dp_9UGvQ_DmUo5Diub0g%40mail.gmail.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHau4yuipzRgx-YmzEzjVJmyuE%3DXkTTZm08pv0-Yp5Lv5Xc97A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHau4yuipzRgx-YmzEzjVJmyuE%3DXkTTZm08pv0-Yp5Lv5Xc97A%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:47am UTC](https://discuss.elastic.co/t/engine-failure-message-outofmemoryerror-unable-to-create-new-native-thread/15717/4 "2017-07-06T01:47:33Z")