Elasticsearch using huge amount of processes

Hello,

we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar)
using multiline{} to our elasticsearch server (version 0.90.2).

After several hours elasticsearch crashes because of not having enough
memory, more precisely, the limit of 10240 open processes for the
elasticsearch user has been reached. No more threads can be created.

The actual setting of ulimit is as follows:

elasticsearch soft nofile 65535
elasticsearch hard nofile 65535
elasticsearch soft nproc 10240
elasticsearch hard nproc 10240

I don't have any clue at the moment, how we can fix this issues. Maybe
someone can help.

Regards
Oliver

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

How many shards do you create per node?
May be you should create less shards or add more machines?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 10 sept. 2013 à 09:32, Oliver tw1nh34d@gmail.com a écrit :

Hello,

we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar) using multiline{} to our elasticsearch server (version 0.90.2).

After several hours elasticsearch crashes because of not having enough memory, more precisely, the limit of 10240 open processes for the elasticsearch user has been reached. No more threads can be created.

The actual setting of ulimit is as follows:

elasticsearch soft nofile 65535
elasticsearch hard nofile 65535
elasticsearch soft nproc 10240
elasticsearch hard nproc 10240

I don't have any clue at the moment, how we can fix this issues. Maybe someone can help.

Regards
Oliver

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi David,

thank you for this fast response. At the moment we are running
elasticsearch with the default settings, means "index.number_of_shards: 5".
We have only one (master) node running.
We are sending the log content (catalina.out) only from one machine. So I
am wondering that this setup creates so many threads, and no thread will
close after some period of time. It is growing to the max ulimit setting
and then crashes.

Regards
Oliver

Am Dienstag, 10. September 2013 09:49:06 UTC+2 schrieb David Pilato:

How many shards do you create per node?
May be you should create less shards or add more machines?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 10 sept. 2013 à 09:32, Oliver <tw1n...@gmail.com <javascript:>> a
écrit :

Hello,

we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar)
using multiline{} to our elasticsearch server (version 0.90.2).

After several hours elasticsearch crashes because of not having enough
memory, more precisely, the limit of 10240 open processes for the
elasticsearch user has been reached. No more threads can be created.

The actual setting of ulimit is as follows:

elasticsearch soft nofile 65535
elasticsearch hard nofile 65535
elasticsearch soft nproc 10240
elasticsearch hard nproc 10240

I don't have any clue at the moment, how we can fix this issues. Maybe
someone can help.

Regards
Oliver

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

And how many index did you create until now? Do you have rolling indexes?

May be you should increase your values?

echo "elasticsearch soft nproc unlimited" | sudo tee -a /etc/security/limits.conf
echo "elasticsearch hard nproc unlimited" | sudo tee -a /etc/security/limits.conf

I think you will need to logout and login again.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 10 sept. 2013 à 11:48, Oliver tw1nh34d@gmail.com a écrit :

Hi David,

thank you for this fast response. At the moment we are running elasticsearch with the default settings, means "index.number_of_shards: 5". We have only one (master) node running.
We are sending the log content (catalina.out) only from one machine. So I am wondering that this setup creates so many threads, and no thread will close after some period of time. It is growing to the max ulimit setting and then crashes.

Regards
Oliver

Am Dienstag, 10. September 2013 09:49:06 UTC+2 schrieb David Pilato:
How many shards do you create per node?
May be you should create less shards or add more machines?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 10 sept. 2013 à 09:32, Oliver tw1n...@gmail.com a écrit :

Hello,

we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar) using multiline{} to our elasticsearch server (version 0.90.2).

After several hours elasticsearch crashes because of not having enough memory, more precisely, the limit of 10240 open processes for the elasticsearch user has been reached. No more threads can be created.

The actual setting of ulimit is as follows:

elasticsearch soft nofile 65535
elasticsearch hard nofile 65535
elasticsearch soft nproc 10240
elasticsearch hard nproc 10240

I don't have any clue at the moment, how we can fix this issues. Maybe someone can help.

Regards
Oliver

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

at the moment, there are these indices written in node 0

ls -la /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
total 60
4 drwxr-xr-x 15 elasticsearch elasticsearch 4096 Sep 10 02:00 .
4 drwxr-xr-x 4 elasticsearch elasticsearch 4096 Sep 9 11:55 ..
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 29 10:50
logstash-2013.08.29
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 30 09:37
logstash-2013.08.30
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54
logstash-2013.08.31
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54
logstash-2013.09.01
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 2 14:03
logstash-2013.09.02
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 3 02:00
logstash-2013.09.03
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 4 14:04
logstash-2013.09.04
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 5 02:00
logstash-2013.09.05
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 6 02:00
logstash-2013.09.06
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08
logstash-2013.09.07
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08
logstash-2013.09.08
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08
logstash-2013.09.09
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 10 02:00
logstash-2013.09.10

We could extend the setting to unlimited, but why will no thread be closed
after a while?
Anyway, I will try your suggestion.

Thanks and Regards
Oliver

Am Dienstag, 10. September 2013 16:13:34 UTC+2 schrieb David Pilato:

And how many index did you create until now? Do you have rolling indexes?

May be you should increase your values?

echo "elasticsearch soft nproc unlimited" | sudo tee -a /etc/security/limits.conf
echo "elasticsearch hard nproc unlimited" | sudo tee -a /etc/security/limits.conf

I think you will need to logout and login again.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 10 sept. 2013 à 11:48, Oliver <tw1n...@gmail.com <javascript:>> a
écrit :

Hi David,

thank you for this fast response. At the moment we are running
elasticsearch with the default settings, means "index.number_of_shards: 5".
We have only one (master) node running.
We are sending the log content (catalina.out) only from one machine. So I
am wondering that this setup creates so many threads, and no thread will
close after some period of time. It is growing to the max ulimit setting
and then crashes.

Regards
Oliver

Am Dienstag, 10. September 2013 09:49:06 UTC+2 schrieb David Pilato:

How many shards do you create per node?
May be you should create less shards or add more machines?

--
David Pilato | Technical Advocate | *Elasticsearch.comhttp://elasticsearch.com/
*
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 10 sept. 2013 à 09:32, Oliver tw1n...@gmail.com a écrit :

Hello,

we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar)
using multiline{} to our elasticsearch server (version 0.90.2).

After several hours elasticsearch crashes because of not having enough
memory, more precisely, the limit of 10240 open processes for the
elasticsearch user has been reached. No more threads can be created.

The actual setting of ulimit is as follows:

elasticsearch soft nofile 65535
elasticsearch hard nofile 65535
elasticsearch soft nproc 10240
elasticsearch hard nproc 10240

I don't have any clue at the moment, how we can fix this issues. Maybe
someone can help.

Regards
Oliver

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

So it means 13 indices with 5 shards. That means 65 Lucene instances running on a single box.
Could you try to close older indices and see how it goes?

http://www.elasticsearch.org/guide/reference/api/admin-indices-open-close/

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 10 sept. 2013 à 16:27, Oliver tw1nh34d@gmail.com a écrit :

at the moment, there are these indices written in node 0

ls -la /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
total 60
4 drwxr-xr-x 15 elasticsearch elasticsearch 4096 Sep 10 02:00 .
4 drwxr-xr-x 4 elasticsearch elasticsearch 4096 Sep 9 11:55 ..
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 29 10:50 logstash-2013.08.29
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 30 09:37 logstash-2013.08.30
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54 logstash-2013.08.31
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54 logstash-2013.09.01
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 2 14:03 logstash-2013.09.02
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 3 02:00 logstash-2013.09.03
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 4 14:04 logstash-2013.09.04
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 5 02:00 logstash-2013.09.05
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 6 02:00 logstash-2013.09.06
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 logstash-2013.09.07
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 logstash-2013.09.08
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08 logstash-2013.09.09
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 10 02:00 logstash-2013.09.10

We could extend the setting to unlimited, but why will no thread be closed after a while?
Anyway, I will try your suggestion.

Thanks and Regards
Oliver

Am Dienstag, 10. September 2013 16:13:34 UTC+2 schrieb David Pilato:
And how many index did you create until now? Do you have rolling indexes?

May be you should increase your values?

echo "elasticsearch soft nproc unlimited" | sudo tee -a /etc/security/limits.conf
echo "elasticsearch hard nproc unlimited" | sudo tee -a /etc/security/limits.conf

I think you will need to logout and login again.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 10 sept. 2013 à 11:48, Oliver tw1n...@gmail.com a écrit :

Hi David,

thank you for this fast response. At the moment we are running elasticsearch with the default settings, means "index.number_of_shards: 5". We have only one (master) node running.
We are sending the log content (catalina.out) only from one machine. So I am wondering that this setup creates so many threads, and no thread will close after some period of time. It is growing to the max ulimit setting and then crashes.

Regards
Oliver

Am Dienstag, 10. September 2013 09:49:06 UTC+2 schrieb David Pilato:
How many shards do you create per node?
May be you should create less shards or add more machines?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 10 sept. 2013 à 09:32, Oliver tw1n...@gmail.com a écrit :

Hello,

we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar) using multiline{} to our elasticsearch server (version 0.90.2).

After several hours elasticsearch crashes because of not having enough memory, more precisely, the limit of 10240 open processes for the elasticsearch user has been reached. No more threads can be created.

The actual setting of ulimit is as follows:

elasticsearch soft nofile 65535
elasticsearch hard nofile 65535
elasticsearch soft nproc 10240
elasticsearch hard nproc 10240

I don't have any clue at the moment, how we can fix this issues. Maybe someone can help.

Regards
Oliver

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I have disabled all indeces except the one from today. Elasticsearch still
uses the maximum of 10240 processes after several hours.

Am Dienstag, 10. September 2013 16:30:12 UTC+2 schrieb David Pilato:

So it means 13 indices with 5 shards. That means 65 Lucene instances
running on a single box.
Could you try to close older indices and see how it goes?

http://www.elasticsearch.org/guide/reference/api/admin-indices-open-close/

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 10 sept. 2013 à 16:27, Oliver <tw1n...@gmail.com <javascript:>> a
écrit :

at the moment, there are these indices written in node 0

ls -la /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
total 60
4 drwxr-xr-x 15 elasticsearch elasticsearch 4096 Sep 10 02:00 .
4 drwxr-xr-x 4 elasticsearch elasticsearch 4096 Sep 9 11:55 ..
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 29 10:50
logstash-2013.08.29
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 30 09:37
logstash-2013.08.30
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54
logstash-2013.08.31
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54
logstash-2013.09.01
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 2 14:03
logstash-2013.09.02
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 3 02:00
logstash-2013.09.03
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 4 14:04
logstash-2013.09.04
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 5 02:00
logstash-2013.09.05
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 6 02:00
logstash-2013.09.06
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08
logstash-2013.09.07
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08
logstash-2013.09.08
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08
logstash-2013.09.09
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 10 02:00
logstash-2013.09.10

We could extend the setting to unlimited, but why will no thread be closed
after a while?
Anyway, I will try your suggestion.

Thanks and Regards
Oliver

Am Dienstag, 10. September 2013 16:13:34 UTC+2 schrieb David Pilato:

And how many index did you create until now? Do you have rolling indexes?

May be you should increase your values?

echo "elasticsearch soft nproc unlimited" | sudo tee -a /etc/security/limits.conf
echo "elasticsearch hard nproc unlimited" | sudo tee -a /etc/security/limits.conf

I think you will need to logout and login again.

--
David Pilato | Technical Advocate | *Elasticsearch.comhttp://elasticsearch.com/
*
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 10 sept. 2013 à 11:48, Oliver tw1n...@gmail.com a écrit :

Hi David,

thank you for this fast response. At the moment we are running
elasticsearch with the default settings, means "index.number_of_shards: 5".
We have only one (master) node running.
We are sending the log content (catalina.out) only from one machine. So I
am wondering that this setup creates so many threads, and no thread will
close after some period of time. It is growing to the max ulimit setting
and then crashes.

Regards
Oliver

Am Dienstag, 10. September 2013 09:49:06 UTC+2 schrieb David Pilato:

How many shards do you create per node?
May be you should create less shards or add more machines?

--
David Pilato | Technical Advocate | *Elasticsearch.comhttp://elasticsearch.com/
*
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr
| @scrutmydocs https://twitter.com/scrutmydocs

Le 10 sept. 2013 à 09:32, Oliver tw1n...@gmail.com a écrit :

Hello,

we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar)
using multiline{} to our elasticsearch server (version 0.90.2).

After several hours elasticsearch crashes because of not having enough
memory, more precisely, the limit of 10240 open processes for the
elasticsearch user has been reached. No more threads can be created.

The actual setting of ulimit is as follows:

elasticsearch soft nofile 65535
elasticsearch hard nofile 65535
elasticsearch soft nproc 10240
elasticsearch hard nproc 10240

I don't have any clue at the moment, how we can fix this issues. Maybe
someone can help.

Regards
Oliver

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

My one machine setup (50GB memory, 4 cores, RHEL) has 3 data indices (15
shards each) and a bunch of marvel indices (~20). I think the same issue is
happening with my setup too.

@David/ Oliver, Did you find the solution to this issue?

Thanks
Yogesh

On Wednesday, September 11, 2013 at 9:21:19 PM UTC+5:30, Oliver wrote:

I have disabled all indeces except the one from today. Elasticsearch still
uses the maximum of 10240 processes after several hours.

Am Dienstag, 10. September 2013 16:30:12 UTC+2 schrieb David Pilato:

So it means 13 indices with 5 shards. That means 65 Lucene instances
running on a single box.
Could you try to close older indices and see how it goes?

http://www.elasticsearch.org/guide/reference/api/admin-indices-open-close/

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 10 sept. 2013 à 16:27, Oliver tw1n...@gmail.com a écrit :

at the moment, there are these indices written in node 0

ls -la /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
total 60
4 drwxr-xr-x 15 elasticsearch elasticsearch 4096 Sep 10 02:00 .
4 drwxr-xr-x 4 elasticsearch elasticsearch 4096 Sep 9 11:55 ..
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 29 10:50
logstash-2013.08.29
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Aug 30 09:37
logstash-2013.08.30
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54
logstash-2013.08.31
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 1 11:54
logstash-2013.09.01
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 2 14:03
logstash-2013.09.02
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 3 02:00
logstash-2013.09.03
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 4 14:04
logstash-2013.09.04
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 5 02:00
logstash-2013.09.05
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 6 02:00
logstash-2013.09.06
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08
logstash-2013.09.07
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08
logstash-2013.09.08
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 9 10:08
logstash-2013.09.09
4 drwxr-xr-x 8 elasticsearch elasticsearch 4096 Sep 10 02:00
logstash-2013.09.10

We could extend the setting to unlimited, but why will no thread be
closed after a while?
Anyway, I will try your suggestion.

Thanks and Regards
Oliver

Am Dienstag, 10. September 2013 16:13:34 UTC+2 schrieb David Pilato:

And how many index did you create until now? Do you have rolling indexes?

May be you should increase your values?

echo "elasticsearch soft nproc unlimited" | sudo tee -a /etc/security/limits.conf
echo "elasticsearch hard nproc unlimited" | sudo tee -a /etc/security/limits.conf

I think you will need to logout and login again.

--
David Pilato | Technical Advocate | Elasticsearch.com
http://elasticsearch.com/

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 10 sept. 2013 à 11:48, Oliver tw1n...@gmail.com a écrit :

Hi David,

thank you for this fast response. At the moment we are running
elasticsearch with the default settings, means "index.number_of_shards: 5".
We have only one (master) node running.
We are sending the log content (catalina.out) only from one machine. So
I am wondering that this setup creates so many threads, and no thread will
close after some period of time. It is growing to the max ulimit setting
and then crashes.

Regards
Oliver

Am Dienstag, 10. September 2013 09:49:06 UTC+2 schrieb David Pilato:

How many shards do you create per node?
May be you should create less shards or add more machines?

--
David Pilato | Technical Advocate | Elasticsearch.com
http://elasticsearch.com/

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 10 sept. 2013 à 09:32, Oliver tw1n...@gmail.com a écrit :

Hello,

we send logfiles (catalina.out) with logstash (version 1.1.13-flatjar)
using multiline{} to our elasticsearch server (version 0.90.2).

After several hours elasticsearch crashes because of not having enough
memory, more precisely, the limit of 10240 open processes for the
elasticsearch user has been reached. No more threads can be created.

The actual setting of ulimit is as follows:

elasticsearch soft nofile 65535
elasticsearch hard nofile 65535
elasticsearch soft nproc 10240
elasticsearch hard nproc 10240

I don't have any clue at the moment, how we can fix this issues. Maybe
someone can help.

Regards
Oliver

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/64e43c93-022f-4211-a7db-99b7a10543c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.