Error : "java.lang.OutOfMemoryError: unable to create new native thread"

Abdurrahim_Eke · August 24, 2010, 8:08am

Hi,

I was testing async indexing to an Elastic Search Node via java
transport api , the ES node have actually 1GB of memory via settings -
Xms1G^ -Xmx1G^ .

ES Node was running on Windows XP ( same problem occurs at Linux
also ) .

The test was like this :

Indexing objects to ES in async mode , objects have only name and
surname .

When i index 50K object everything is fine,
When i index 100K objects i see the OutOfMemoryError in the logs,
It seems on some limit the error occurs.
When the error occurs the heap size is around 250MB , means there is
still 750MB free memory to use.

I did some research and found that the reason may be the stack size
from http://candrews.integralblue.com/2009/01/preventing-outofmemoryerror-native-thread/

For 100K objects I have changed elasticsearch.bat tried -Xss256k ,
sometimes it gives no error but sometimes the error occurs again.

Please advise on this: is this a bug, if this is not a bug how should
we overcome the problem.

The error from ES logs :

[18:26:53,392][WARN ][transport.netty ] [Immortus] Exception
caught on netty layer [[id: 0x01bca1c3, /192.168.1.107:27288 => /
192.168.1.107:9350]] java.lang.OutOfMemoryError: unable to create new
native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:
727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
657)
at
org.elasticsearch.threadpool.support.AbstractThreadPool.execute(AbstractThreadPool.java:
146)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction
$AsyncShardOperationAction.start(TransportShardReplicationOperationAction.java:
276)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction
$AsyncShardOperationAction.start(TransportShardReplicationOperationAction.java:
240)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction.doExecute(TransportShardReplicationOperationAction.java:
93)
at
org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:
101)
at
org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:
57)
at
org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:
54)
...

Regards,
Abdurrahim Eke
www.ifountain.com

kimchy · August 24, 2010, 11:00am

By default, elasticsearch uses a cache thread pool, meaning it has an
unbounded thread pool (though thread are reused). You can switch to a thread
pool that has limit on the number of threads, and then choose if the request
should be blocked till there is an available thread, or queued for later
execution (blocking is probably better).

In any case, the end result will be the same, just a different problem. With
a cached thread pool, you might run out of threads. With blocking based one,
you will get a failure if the blocking times out. This will happen if you
"hammer" elasticsearch in a way that it can't process requests fast enough.
In the async test, if you don't wait for the responses and just push them,
you will get into this state (indexing takes its time, even though its
small). You will need to throttle the indexing operation, even if its async.

-shay.banon

On Tue, Aug 24, 2010 at 11:08 AM, Abdurrahim Eke ekelog@gmail.com wrote:

Hi,

I was testing async indexing to an Elastic Search Node via java
transport api , the ES node have actually 1GB of memory via settings -
Xms1G^ -Xmx1G^ .

ES Node was running on Windows XP ( same problem occurs at Linux
also ) .

The test was like this :

Indexing objects to ES in async mode , objects have only name and
surname .

When i index 50K object everything is fine,
When i index 100K objects i see the OutOfMemoryError in the logs,
It seems on some limit the error occurs.
When the error occurs the heap size is around 250MB , means there is
still 750MB free memory to use.

I did some research and found that the reason may be the stack size
from
Preventing java.lang.OutOfMemoryError: unable to create new native thread – Craig Andrews

For 100K objects I have changed elasticsearch.bat tried -Xss256k ,
sometimes it gives no error but sometimes the error occurs again.

Please advise on this: is this a bug, if this is not a bug how should
we overcome the problem.

The error from ES logs :

[18:26:53,392][WARN ][transport.netty ] [Immortus] Exception
caught on netty layer [[id: 0x01bca1c3, /192.168.1.107:27288 => /
192.168.1.107:9350]] java.lang.OutOfMemoryError: unable to create new
native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:597)
at

java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:
727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:
657)
at

org.elasticsearch.threadpool.support.AbstractThreadPool.execute(AbstractThreadPool.java:
146)
at

org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction

$AsyncShardOperationAction.start(TransportShardReplicationOperationAction.java:
276)
at

org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction

$AsyncShardOperationAction.start(TransportShardReplicationOperationAction.java:
240)
at

org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction.doExecute(TransportShardReplicationOperationAction.java:
93)
at

org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:
101)
at

org.elasticsearch.action.index.TransportIndexAction.doExecute(TransportIndexAction.java:
57)
at
org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:
54)
...

Regards,
Abdurrahim Eke
www.ifountain.com

Abdurrahim_Eke · August 25, 2010, 2:24pm

Hi Shay,

Thanks for your suggestions, we actually tried scaling threadpool
module , but it seems there is a bug and Elasticsearch does not read
the configuration .
I have created an issue on ThreadPool Module configuration is not parsed, ThreadPoolModule.configure() is wrong · Issue #335 · elastic/elasticsearch · GitHub
, a copy of the issue here

With the following node yml configuration ThreadPool settings does not
change :
threadpool:
type: scaling
scaling:
min: 1
max: 5
scheduled_size: 5

The thread pool type is still cached from : http://localhost:9200/_cluster/nodes

After digging the source code we found that :
ThreadPoolModule.configure() is wrong :
Class<? extends Module> moduleClass =
settings.getAsClass("transport.type", CachedThreadPoolModule.class,
"org.elasticsearch.threadpool.", "ThreadPoolModule");

i guess it should be as : settings.getAsClass("threadpool.type"

Regards,
Abdurrahim Eke
www.ifountain.com

On Aug 24, 2:00 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

By default, elasticsearch uses a cache thread pool, meaning it has an
unbounded thread pool (though thread are reused). You can switch to a thread
pool that has limit on the number of threads, and then choose if the request
should be blocked till there is an available thread, or queued for later
execution (blocking is probably better).

In any case, the end result will be the same, just a different problem. With
a cached thread pool, you might run out of threads. With blocking based one,
you will get a failure if the blocking times out. This will happen if you
"hammer" elasticsearch in a way that it can't process requests fast enough.
In the async test, if you don't wait for the responses and just push them,
you will get into this state (indexing takes its time, even though its
small). You will need to throttle the indexing operation, even if its async.

-shay.banon

kimchy · August 25, 2010, 2:26pm

Yea, I already fixed it in master ;). I suggest using the blocking one
instead of the scaling one.

-shay.banon

On Wed, Aug 25, 2010 at 5:24 PM, Abdurrahim Eke ekelog@gmail.com wrote:

Hi Shay,

Thanks for your suggestions, we actually tried scaling threadpool
module , but it seems there is a bug and Elasticsearch does not read
the configuration .
I have created an issue on
ThreadPool Module configuration is not parsed, ThreadPoolModule.configure() is wrong · Issue #335 · elastic/elasticsearch · GitHub
, a copy of the issue here

With the following node yml configuration ThreadPool settings does not
change :
threadpool:
type: scaling
scaling:
min: 1
max: 5
scheduled_size: 5

The thread pool type is still cached from :
http://localhost:9200/_cluster/nodes

After digging the source code we found that :
ThreadPoolModule.configure() is wrong :
Class<? extends Module> moduleClass =
settings.getAsClass("transport.type", CachedThreadPoolModule.class,
"org.elasticsearch.threadpool.", "ThreadPoolModule");

i guess it should be as : settings.getAsClass("threadpool.type"

Regards,
Abdurrahim Eke
www.ifountain.com

On Aug 24, 2:00 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

By default, elasticsearch uses a cache thread pool, meaning it has an
unbounded thread pool (though thread are reused). You can switch to a
thread
pool that has limit on the number of threads, and then choose if the
request
should be blocked till there is an available thread, or queued for later
execution (blocking is probably better).

In any case, the end result will be the same, just a different problem.
With
a cached thread pool, you might run out of threads. With blocking based
one,
you will get a failure if the blocking times out. This will happen if you
"hammer" elasticsearch in a way that it can't process requests fast
enough.
In the async test, if you don't wait for the responses and just push
them,
you will get into this state (indexing takes its time, even though its
small). You will need to throttle the indexing operation, even if its
async.

-shay.banon

mohammad · February 20, 2014, 11:11am

Hello everyone,

I am a little new one elastic search,
Actually i am implementing somewhat of the following :
example (http://java.dzone.com/articles/elasticsearch-java-api)

...
TermsFacetBuilder fba = FacetBuilders.termsFacet("userId").field("userId");
SearchRequestBuilder srb1a = client.prepareSearch().setIndices(index).addFacet(fba).addSort(SortBuilders.fieldSort("dateTimeStamp").order(SortOrder.DESC));
AndFilterBuilder myFiltersa = FilterBuilders.andFilter();
myFiltersa.add(FilterBuilders.rangeFilter("dateTimeStamp").from(dureeFrom).to(dureeTo));
FilterBuilder fbBuildera = FilterBuilders.andFilter(myFiltersa);
FilteredQueryBuilder qa = QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),fbBuildera);
SearchResponse sra = srb1a.setQuery(qa).execute().actionGet();
...

Actually , i am running this part of codes for every 2 mins, (i have to retrieve data in real Time).
However , after a certain time i get the following errors:

Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at org.elasticsearch.threadpool.ThreadPool.(ThreadPool.java:132)
at sun.reflect.GeneratedConstructorAccessor83.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)..........

I have searched a lot and found some sites whereby users have suggested modifying the elasticsearch.yml

I am actually using eclipse kepler and imported elastic search 0.90.0 as jar. There has not been any installation of elastic search.
As a matter of fact i did not find any file named elasticsearch.yml on my workspace or in the jar files.
Can you help me on this issue because this is becoming more and mre frustrating.
any workaround
thanks in advance.