# Elasticsearch using huge amount of processes **URL:** \ **Category:** Elasticsearch\ **Created:** [September 10, 2013, 7:32am UTC](https://discuss.elastic.co/t/elasticsearch-using-huge-amount-of-processes/13531 "2013-09-10T07:32:34Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Oliver1](https://avatars.discourse-cdn.com/v4/letter/o/eb9ed0/32.png) [@Oliver1](https://discuss.elastic.co/u/Oliver1)\ **Post date:** [September 10, 2013, 7:32am UTC](https://discuss.elastic.co/t/elasticsearch-using-huge-amount-of-processes/13531/1 "2013-09-10T07:32:34Z")
Hello, we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar) using multiline{} to our elasticsearch server (version 0.90.2). After several hours elasticsearch crashes because of not having enough memory, more precisely, the limit of 10240 open processes for the elasticsearch user has been reached. No more threads can be created. The actual setting of ulimit is as follows: elasticsearch soft nofile 65535 elasticsearch hard nofile 65535 elasticsearch soft nproc 10240 elasticsearch hard nproc 10240 I don't have any clue at the moment, how we can fix this issues. Maybe someone can help. Regards Oliver -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 10, 2013, 7:49am UTC](https://discuss.elastic.co/t/elasticsearch-using-huge-amount-of-processes/13531/2 "2013-09-10T07:49:06Z")
How many shards do you create per node? May be you should create less shards or add more machines? -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 10 sept. 2013 à 09:32, Oliver [tw1nh34d@gmail.com](mailto:tw1nh34d@gmail.com) a écrit : > Hello, > > we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar) using multiline{} to our elasticsearch server (version 0.90.2). > > After several hours elasticsearch crashes because of not having enough memory, more precisely, the limit of 10240 open processes for the elasticsearch user has been reached. No more threads can be created. > > The actual setting of ulimit is as follows: > > elasticsearch soft nofile 65535 > elasticsearch hard nofile 65535 > elasticsearch soft nproc 10240 > elasticsearch hard nproc 10240 > > I don't have any clue at the moment, how we can fix this issues. Maybe someone can help. > > Regards > Oliver > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Oliver1](https://avatars.discourse-cdn.com/v4/letter/o/eb9ed0/32.png) [@Oliver1](https://discuss.elastic.co/u/Oliver1)\ **Post date:** [September 10, 2013, 9:48am UTC](https://discuss.elastic.co/t/elasticsearch-using-huge-amount-of-processes/13531/3 "2013-09-10T09:48:05Z")
Hi David, thank you for this fast response. At the moment we are running elasticsearch with the default settings, means "index.number\_of\_shards: 5". We have only one (master) node running. We are sending the log content (catalina.out) only from one machine. So I am wondering that this setup creates so many threads, and no thread will close after some period of time. It is growing to the max ulimit setting and then crashes. Regards Oliver Am Dienstag, 10. September 2013 09:49:06 UTC+2 schrieb David Pilato: > How many shards do you create per node? > May be you should create less shards or add more machines? > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > Le 10 sept. 2013 à 09:32, Oliver \<[tw1n...@gmail.com](mailto:tw1n...@gmail.com) \\> a > écrit : > > Hello, > > we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar) > using multiline{} to our elasticsearch server (version 0.90.2). > > After several hours elasticsearch crashes because of not having enough > memory, more precisely, the limit of 10240 open processes for the > elasticsearch user has been reached. No more threads can be created. > > The actual setting of ulimit is as follows: > > elasticsearch soft nofile 65535 > elasticsearch hard nofile 65535 > elasticsearch soft nproc 10240 > elasticsearch hard nproc 10240 > > I don't have any clue at the moment, how we can fix this issues. Maybe > someone can help. > > Regards > Oliver > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 10, 2013, 2:13pm UTC](https://discuss.elastic.co/t/elasticsearch-using-huge-amount-of-processes/13531/4 "2013-09-10T14:13:34Z")
And how many index did you create until now? Do you have rolling indexes? May be you should increase your values? echo "elasticsearch soft nproc unlimited" | sudo tee -a /etc/security/limits.conf echo "elasticsearch hard nproc unlimited" | sudo tee -a /etc/security/limits.conf I think you will need to logout and login again. -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 10 sept. 2013 à 11:48, Oliver [tw1nh34d@gmail.com](mailto:tw1nh34d@gmail.com) a écrit : > Hi David, > > thank you for this fast response. At the moment we are running elasticsearch with the default settings, means "index.number\_of\_shards: 5". We have only one (master) node running. > We are sending the log content (catalina.out) only from one machine. So I am wondering that this setup creates so many threads, and no thread will close after some period of time. It is growing to the max ulimit setting and then crashes. > > Regards > Oliver > > Am Dienstag, 10. September 2013 09:49:06 UTC+2 schrieb David Pilato: > How many shards do you create per node? > May be you should create less shards or add more machines? > > -- > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) > @dadoonet | @elasticsearchfr | @scrutmydocs > > Le 10 sept. 2013 à 09:32, Oliver [tw1n...@gmail.com](mailto:tw1n...@gmail.com) a écrit : > > > Hello, > > > > we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar) using multiline{} to our elasticsearch server (version 0.90.2). > > > > After several hours elasticsearch crashes because of not having enough memory, more precisely, the limit of 10240 open processes for the elasticsearch user has been reached. No more threads can be created. > > > > The actual setting of ulimit is as follows: > > > > elasticsearch soft nofile 65535 > > elasticsearch hard nofile 65535 > > elasticsearch soft nproc 10240 > > elasticsearch hard nproc 10240 > > > > I don't have any clue at the moment, how we can fix this issues. Maybe someone can help. > > > > Regards > > Oliver > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Oliver1](https://avatars.discourse-cdn.com/v4/letter/o/eb9ed0/32.png) [@Oliver1](https://discuss.elastic.co/u/Oliver1)\ **Post date:** [September 10, 2013, 2:27pm UTC](https://discuss.elastic.co/t/elasticsearch-using-huge-amount-of-processes/13531/5 "2013-09-10T14:27:20Z")
at the moment, there are these indices written in node 0 ls -la /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ total 60 4 drwxr-xr-x 15 elasticsearch elasticsearch 4096 Sep 10 02:00 . 4 drwxr-xr-x 4 elasticsearch elasticsearch 4096 Sep 9 11:55 .. 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 29 10:50 logstash-2013.08.29 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 30 09:37 logstash-2013.08.30 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54 logstash-2013.08.31 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54 logstash-2013.09.01 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 2 14:03 logstash-2013.09.02 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 3 02:00 logstash-2013.09.03 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 4 14:04 logstash-2013.09.04 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 5 02:00 logstash-2013.09.05 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 6 02:00 logstash-2013.09.06 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 logstash-2013.09.07 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 logstash-2013.09.08 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 logstash-2013.09.09 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 10 02:00 logstash-2013.09.10 We could extend the setting to unlimited, but why will no thread be closed after a while? Anyway, I will try your suggestion. Thanks and Regards Oliver Am Dienstag, 10. September 2013 16:13:34 UTC+2 schrieb David Pilato: > And how many index did you create until now? Do you have rolling indexes? > > May be you should increase your values? > > echo "elasticsearch soft nproc unlimited" | sudo tee -a /etc/security/limits.conf > echo "elasticsearch hard nproc unlimited" | sudo tee -a /etc/security/limits.conf > > I think you will need to logout and login again. > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > Le 10 sept. 2013 à 11:48, Oliver \<[tw1n...@gmail.com](mailto:tw1n...@gmail.com) \\> a > écrit : > > Hi David, > > thank you for this fast response. At the moment we are running > elasticsearch with the default settings, means "index.number\_of\_shards: 5". > We have only one (master) node running. > We are sending the log content (catalina.out) only from one machine. So I > am wondering that this setup creates so many threads, and no thread will > close after some period of time. It is growing to the max ulimit setting > and then crashes. > > Regards > Oliver > > Am Dienstag, 10. September 2013 09:49:06 UTC+2 schrieb David Pilato: > > > How many shards do you create per node? > > May be you should create less shards or add more machines? > > > > -- > > _David Pilato_ | _Technical Advocate_ | \*[Elasticsearch.com](http://Elasticsearch.com)[http://elasticsearch.com/](http://elasticsearch.com/) > > \* > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > > > Le 10 sept. 2013 à 09:32, Oliver [tw1n...@gmail.com](mailto:tw1n...@gmail.com) a écrit : > > > > Hello, > > > > we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar) > > using multiline{} to our elasticsearch server (version 0.90.2). > > > > After several hours elasticsearch crashes because of not having enough > > memory, more precisely, the limit of 10240 open processes for the > > elasticsearch user has been reached. No more threads can be created. > > > > The actual setting of ulimit is as follows: > > > > elasticsearch soft nofile 65535 > > elasticsearch hard nofile 65535 > > elasticsearch soft nproc 10240 > > elasticsearch hard nproc 10240 > > > > I don't have any clue at the moment, how we can fix this issues. Maybe > > someone can help. > > > > Regards > > Oliver > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 10, 2013, 2:30pm UTC](https://discuss.elastic.co/t/elasticsearch-using-huge-amount-of-processes/13531/6 "2013-09-10T14:30:12Z")
So it means 13 indices with 5 shards. That means 65 Lucene instances running on a single box. Could you try to close older indices and see how it goes? > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet | @elasticsearchfr | @scrutmydocs Le 10 sept. 2013 à 16:27, Oliver [tw1nh34d@gmail.com](mailto:tw1nh34d@gmail.com) a écrit : > at the moment, there are these indices written in node 0 > > ls -la /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ > total 60 > 4 drwxr-xr-x 15 elasticsearch elasticsearch 4096 Sep 10 02:00 . > 4 drwxr-xr-x 4 elasticsearch elasticsearch 4096 Sep 9 11:55 .. > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 29 10:50 logstash-2013.08.29 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 30 09:37 logstash-2013.08.30 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54 logstash-2013.08.31 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54 logstash-2013.09.01 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 2 14:03 logstash-2013.09.02 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 3 02:00 logstash-2013.09.03 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 4 14:04 logstash-2013.09.04 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 5 02:00 logstash-2013.09.05 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 6 02:00 logstash-2013.09.06 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 logstash-2013.09.07 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 logstash-2013.09.08 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 logstash-2013.09.09 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 10 02:00 logstash-2013.09.10 > > We could extend the setting to unlimited, but why will no thread be closed after a while? > Anyway, I will try your suggestion. > > Thanks and Regards > Oliver > > Am Dienstag, 10. September 2013 16:13:34 UTC+2 schrieb David Pilato: > And how many index did you create until now? Do you have rolling indexes? > > May be you should increase your values? > > echo "elasticsearch soft nproc unlimited" | sudo tee -a /etc/security/limits.conf > echo "elasticsearch hard nproc unlimited" | sudo tee -a /etc/security/limits.conf > > I think you will need to logout and login again. > > -- > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) > @dadoonet | @elasticsearchfr | @scrutmydocs > > Le 10 sept. 2013 à 11:48, Oliver [tw1n...@gmail.com](mailto:tw1n...@gmail.com) a écrit : > > > Hi David, > > > > thank you for this fast response. At the moment we are running elasticsearch with the default settings, means "index.number\_of\_shards: 5". We have only one (master) node running. > > We are sending the log content (catalina.out) only from one machine. So I am wondering that this setup creates so many threads, and no thread will close after some period of time. It is growing to the max ulimit setting and then crashes. > > > > Regards > > Oliver > > > > Am Dienstag, 10. September 2013 09:49:06 UTC+2 schrieb David Pilato: > > How many shards do you create per node? > > May be you should create less shards or add more machines? > > > > -- > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) > > @dadoonet | @elasticsearchfr | @scrutmydocs > > > > Le 10 sept. 2013 à 09:32, Oliver [tw1n...@gmail.com](mailto:tw1n...@gmail.com) a écrit : > > > > > Hello, > > > > > > we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar) using multiline{} to our elasticsearch server (version 0.90.2). > > > > > > After several hours elasticsearch crashes because of not having enough memory, more precisely, the limit of 10240 open processes for the elasticsearch user has been reached. No more threads can be created. > > > > > > The actual setting of ulimit is as follows: > > > > > > elasticsearch soft nofile 65535 > > > elasticsearch hard nofile 65535 > > > elasticsearch soft nproc 10240 > > > elasticsearch hard nproc 10240 > > > > > > I don't have any clue at the moment, how we can fix this issues. Maybe someone can help. > > > > > > Regards > > > Oliver > > > > > > -- > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Oliver1](https://avatars.discourse-cdn.com/v4/letter/o/eb9ed0/32.png) [@Oliver1](https://discuss.elastic.co/u/Oliver1)\ **Post date:** [September 11, 2013, 3:51pm UTC](https://discuss.elastic.co/t/elasticsearch-using-huge-amount-of-processes/13531/7 "2013-09-11T15:51:19Z")
I have disabled all indeces except the one from today. Elasticsearch still uses the maximum of 10240 processes after several hours. Am Dienstag, 10. September 2013 16:30:12 UTC+2 schrieb David Pilato: > So it means 13 indices with 5 shards. That means 65 Lucene instances > running on a single box. > Could you try to close older indices and see how it goes? > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-open-close/) > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > Le 10 sept. 2013 à 16:27, Oliver \<[tw1n...@gmail.com](mailto:tw1n...@gmail.com) \\> a > écrit : > > at the moment, there are these indices written in node 0 > > ls -la /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ > total 60 > 4 drwxr-xr-x 15 elasticsearch elasticsearch 4096 Sep 10 02:00 . > 4 drwxr-xr-x 4 elasticsearch elasticsearch 4096 Sep 9 11:55 .. > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 29 10:50 > logstash-2013.08.29 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 30 09:37 > logstash-2013.08.30 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54 > logstash-2013.08.31 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54 > logstash-2013.09.01 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 2 14:03 > logstash-2013.09.02 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 3 02:00 > logstash-2013.09.03 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 4 14:04 > logstash-2013.09.04 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 5 02:00 > logstash-2013.09.05 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 6 02:00 > logstash-2013.09.06 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 > logstash-2013.09.07 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 > logstash-2013.09.08 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 > logstash-2013.09.09 > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 10 02:00 > logstash-2013.09.10 > > We could extend the setting to unlimited, but why will no thread be closed > after a while? > Anyway, I will try your suggestion. > > Thanks and Regards > Oliver > > Am Dienstag, 10. September 2013 16:13:34 UTC+2 schrieb David Pilato: > > > And how many index did you create until now? Do you have rolling indexes? > > > > May be you should increase your values? > > > > echo "elasticsearch soft nproc unlimited" | sudo tee -a /etc/security/limits.conf > > echo "elasticsearch hard nproc unlimited" | sudo tee -a /etc/security/limits.conf > > > > I think you will need to logout and login again. > > > > -- > > _David Pilato_ | _Technical Advocate_ | \*[Elasticsearch.com](http://Elasticsearch.com)[http://elasticsearch.com/](http://elasticsearch.com/) > > \* > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > > > Le 10 sept. 2013 à 11:48, Oliver [tw1n...@gmail.com](mailto:tw1n...@gmail.com) a écrit : > > > > Hi David, > > > > thank you for this fast response. At the moment we are running > > elasticsearch with the default settings, means "index.number\_of\_shards: 5". > > We have only one (master) node running. > > We are sending the log content (catalina.out) only from one machine. So I > > am wondering that this setup creates so many threads, and no thread will > > close after some period of time. It is growing to the max ulimit setting > > and then crashes. > > > > Regards > > Oliver > > > > Am Dienstag, 10. September 2013 09:49:06 UTC+2 schrieb David Pilato: > > > > > How many shards do you create per node? > > > May be you should create less shards or add more machines? > > > > > > -- > > > _David Pilato_ | _Technical Advocate_ | \*[Elasticsearch.com](http://Elasticsearch.com)[http://elasticsearch.com/](http://elasticsearch.com/) > > > \* > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr[https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) > > > | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > > > > > Le 10 sept. 2013 à 09:32, Oliver [tw1n...@gmail.com](mailto:tw1n...@gmail.com) a écrit : > > > > > > Hello, > > > > > > we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar) > > > using multiline{} to our elasticsearch server (version 0.90.2). > > > > > > After several hours elasticsearch crashes because of not having enough > > > memory, more precisely, the limit of 10240 open processes for the > > > elasticsearch user has been reached. No more threads can be created. > > > > > > The actual setting of ulimit is as follows: > > > > > > elasticsearch soft nofile 65535 > > > elasticsearch hard nofile 65535 > > > elasticsearch soft nproc 10240 > > > elasticsearch hard nproc 10240 > > > > > > I don't have any clue at the moment, how we can fix this issues. Maybe > > > someone can help. > > > > > > Regards > > > Oliver > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Yogesh\_Kansal](https://avatars.discourse-cdn.com/v4/letter/y/57b2e6/32.png) [@Yogesh\_Kansal](https://discuss.elastic.co/u/Yogesh_Kansal)\ **Post date:** [March 26, 2015, 8:10am UTC](https://discuss.elastic.co/t/elasticsearch-using-huge-amount-of-processes/13531/8 "2015-03-26T08:10:40Z")
My one machine setup (50GB memory, 4 cores, RHEL) has 3 data indices (15 shards each) and a bunch of marvel indices (~20). I think the same issue is happening with my setup too. @David/ Oliver, Did you find the solution to this issue? Thanks Yogesh On Wednesday, September 11, 2013 at 9:21:19 PM UTC+5:30, Oliver wrote: > I have disabled all indeces except the one from today. Elasticsearch still > uses the maximum of 10240 processes after several hours. > > Am Dienstag, 10. September 2013 16:30:12 UTC+2 schrieb David Pilato: > > > So it means 13 indices with 5 shards. That means 65 Lucene instances > > running on a single box. > > Could you try to close older indices and see how it goes? > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-open-close/) > > > > -- > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com) > > [http://Elasticsearch.com](http://Elasticsearch.com)_ > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs > > [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > > > Le 10 sept. 2013 à 16:27, Oliver [tw1n...@gmail.com](mailto:tw1n...@gmail.com) a écrit : > > > > at the moment, there are these indices written in node 0 > > > > ls -la /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ > > total 60 > > 4 drwxr-xr-x 15 elasticsearch elasticsearch 4096 Sep 10 02:00 . > > 4 drwxr-xr-x 4 elasticsearch elasticsearch 4096 Sep 9 11:55 .. > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 29 10:50 > > logstash-2013.08.29 > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 30 09:37 > > logstash-2013.08.30 > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54 > > logstash-2013.08.31 > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54 > > logstash-2013.09.01 > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 2 14:03 > > logstash-2013.09.02 > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 3 02:00 > > logstash-2013.09.03 > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 4 14:04 > > logstash-2013.09.04 > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 5 02:00 > > logstash-2013.09.05 > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 6 02:00 > > logstash-2013.09.06 > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 > > logstash-2013.09.07 > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 > > logstash-2013.09.08 > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 > > logstash-2013.09.09 > > 4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 10 02:00 > > logstash-2013.09.10 > > > > We could extend the setting to unlimited, but why will no thread be > > closed after a while? > > Anyway, I will try your suggestion. > > > > Thanks and Regards > > Oliver > > > > Am Dienstag, 10. September 2013 16:13:34 UTC+2 schrieb David Pilato: > > > > > And how many index did you create until now? Do you have rolling indexes? > > > > > > May be you should increase your values? > > > > > > echo "elasticsearch soft nproc unlimited" | sudo tee -a /etc/security/limits.conf > > > echo "elasticsearch hard nproc unlimited" | sudo tee -a /etc/security/limits.conf > > > > > > I think you will need to logout and login again. > > > > > > -- > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com) > > > [http://elasticsearch.com/](http://elasticsearch.com/)_ > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr > > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs > > > [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > > > > > Le 10 sept. 2013 à 11:48, Oliver [tw1n...@gmail.com](mailto:tw1n...@gmail.com) a écrit : > > > > > > Hi David, > > > > > > thank you for this fast response. At the moment we are running > > > elasticsearch with the default settings, means "index.number\_of\_shards: 5". > > > We have only one (master) node running. > > > We are sending the log content (catalina.out) only from one machine. So > > > I am wondering that this setup creates so many threads, and no thread will > > > close after some period of time. It is growing to the max ulimit setting > > > and then crashes. > > > > > > Regards > > > Oliver > > > > > > Am Dienstag, 10. September 2013 09:49:06 UTC+2 schrieb David Pilato: > > > > > > > How many shards do you create per node? > > > > May be you should create less shards or add more machines? > > > > > > > > -- > > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com) > > > > [http://elasticsearch.com/](http://elasticsearch.com/)_ > > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr > > > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs > > > > [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > > > > > > > Le 10 sept. 2013 à 09:32, Oliver [tw1n...@gmail.com](mailto:tw1n...@gmail.com) a écrit : > > > > > > > > Hello, > > > > > > > > we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar) > > > > using multiline{} to our elasticsearch server (version 0.90.2). > > > > > > > > After several hours elasticsearch crashes because of not having enough > > > > memory, more precisely, the limit of 10240 open processes for the > > > > elasticsearch user has been reached. No more threads can be created. > > > > > > > > The actual setting of ulimit is as follows: > > > > > > > > elasticsearch soft nofile 65535 > > > > elasticsearch hard nofile 65535 > > > > elasticsearch soft nproc 10240 > > > > elasticsearch hard nproc 10240 > > > > > > > > I don't have any clue at the moment, how we can fix this issues. Maybe > > > > someone can help. > > > > > > > > Regards > > > > Oliver > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/64e43c93-022f-4211-a7db-99b7a10543c9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/64e43c93-022f-4211-a7db-99b7a10543c9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:23am UTC](https://discuss.elastic.co/t/elasticsearch-using-huge-amount-of-processes/13531/9 "2017-07-06T00:23:49Z")