Elasticsearch client is giving OutOfMemoryError once connection is lost to elaticsearch server

Elastic Stack Elasticsearch
1

Hi,

I'm using elasticsearch using TransportClient for multiple operation. The
issue I'm facing now is if my es server goes down my client side app
getting OutOfMemoryError. Getting the below exception. I had to restart my
tomcat every time after this to make my application up. Can some one please
suggest how to prevent this.

Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [spring] in context with path
[/aricloud] threw exception [Handler processing failed; nested exception is
java.lang.OutOfMemoryError: unable to create new native thread] with root
cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:349)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:100)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:52)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)

Thanks,
Subhadip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6930af3b-13ad-476c-82cc-7c8d792dea67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Other members can correct me if I'm wrong, but I notice that when you lose
connection with the server, the transportclient queues retries of whatever
operations you try to execute, and it starts to queue listeners into a
'generic' threadpool (which I read somewhere that it was unbounded). We've
seen this problem when we thrash ES until it eventually stops responding,
and our bulk requests start to back up and eventually cause the application
to halt due to OOM.

I don't know exactly what your application is doing when your ES node(s) go
down, but perhaps you can proactively stop requests to ES servers once your
application sees the no node exception error (which you should get when ES
goes down). You could also close the transportclient and shutdown its
threadpool and reconnect/instantiate after a timed delay to clean up
whatever is floating around in the transportclient. We have been able to
solve most of our native thread issues by protecting our use of
transportclient and doing a soft restart of this client.

On Saturday, January 10, 2015 at 9:29:56 AM UTC-8, Subhadip Bagui wrote:

Hi,

I'm using elasticsearch using TransportClient for multiple operation. The
issue I'm facing now is if my es server goes down my client side app
getting OutOfMemoryError. Getting the below exception. I had to restart my
tomcat every time after this to make my application up. Can some one please
suggest how to prevent this.

Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [spring] in context with path
[/aricloud] threw exception [Handler processing failed; nested exception is
java.lang.OutOfMemoryError: unable to create new native thread] with root
cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:349)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:100)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:52)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)

Thanks,
Subhadip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Ed,

soft restart as in client.close() ? I did that as well, does not seem to
work, hence I resolve to tomcat restart instead...

Jason

On Sun, Jan 11, 2015 at 3:03 PM, Ed Kim edkim81@gmail.com wrote:

Other members can correct me if I'm wrong, but I notice that when you lose
connection with the server, the transportclient queues retries of whatever
operations you try to execute, and it starts to queue listeners into a
'generic' threadpool (which I read somewhere that it was unbounded). We've
seen this problem when we thrash ES until it eventually stops responding,
and our bulk requests start to back up and eventually cause the application
to halt due to OOM.

I don't know exactly what your application is doing when your ES node(s)
go down, but perhaps you can proactively stop requests to ES servers once
your application sees the no node exception error (which you should get
when ES goes down). You could also close the transportclient and shutdown
its threadpool and reconnect/instantiate after a timed delay to clean up
whatever is floating around in the transportclient. We have been able to
solve most of our native thread issues by protecting our use of
transportclient and doing a soft restart of this client.

On Saturday, January 10, 2015 at 9:29:56 AM UTC-8, Subhadip Bagui wrote:

Hi,

I'm using elasticsearch using TransportClient for multiple operation. The
issue I'm facing now is if my es server goes down my client side app
getting OutOfMemoryError. Getting the below exception. I had to restart my
tomcat every time after this to make my application up. Can some one please
suggest how to prevent this.

Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve
invoke
SEVERE: Servlet.service() for servlet [spring] in context with path
[/aricloud] threw exception [Handler processing failed; nested exception is
java.lang.OutOfMemoryError: unable to create new native thread] with root
cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.util.concurrent.ThreadPoolExecutor.addThread(
ThreadPoolExecutor.java:681)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(
ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(
ThreadPoolExecutor.java:655)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.
openSelector(AbstractNioSelector.java:349)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.(AbstractNioSelector.java:100)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.(AbstractNioWorker.java:52)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.(NioWorker.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorkerPool.createWorker(NioWorkerPool.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorkerPool.createWorker(NioWorkerPool.java:28)

Thanks,
Subhadip

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4ityTv%2B2-tQLrqWQetKfL0juxDVqhsJeNbYQkZbzr6kjtDg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

Yes, but in addition I would also call client.threadPool().shutdown() to
make sure you free up the threadpool.

On Sunday, January 11, 2015 at 7:22:20 PM UTC-8, Jason Wee wrote:

Hi Ed,

soft restart as in client.close() ? I did that as well, does not seem to
work, hence I resolve to tomcat restart instead...

Jason

On Sun, Jan 11, 2015 at 3:03 PM, Ed Kim <edk...@gmail.com <javascript:>>
wrote:

Other members can correct me if I'm wrong, but I notice that when you
lose connection with the server, the transportclient queues retries of
whatever operations you try to execute, and it starts to queue listeners
into a 'generic' threadpool (which I read somewhere that it was unbounded).
We've seen this problem when we thrash ES until it eventually stops
responding, and our bulk requests start to back up and eventually cause the
application to halt due to OOM.

I don't know exactly what your application is doing when your ES node(s)
go down, but perhaps you can proactively stop requests to ES servers once
your application sees the no node exception error (which you should get
when ES goes down). You could also close the transportclient and shutdown
its threadpool and reconnect/instantiate after a timed delay to clean up
whatever is floating around in the transportclient. We have been able to
solve most of our native thread issues by protecting our use of
transportclient and doing a soft restart of this client.

On Saturday, January 10, 2015 at 9:29:56 AM UTC-8, Subhadip Bagui wrote:

Hi,

I'm using elasticsearch using TransportClient for multiple operation.
The issue I'm facing now is if my es server goes down my client side app
getting OutOfMemoryError. Getting the below exception. I had to restart my
tomcat every time after this to make my application up. Can some one please
suggest how to prevent this.

Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve
invoke
SEVERE: Servlet.service() for servlet [spring] in context with path
[/aricloud] threw exception [Handler processing failed; nested exception is
java.lang.OutOfMemoryError: unable to create new native thread] with root
cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.util.concurrent.ThreadPoolExecutor.addThread(
ThreadPoolExecutor.java:681)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(
ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(
ThreadPoolExecutor.java:655)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.openSelector(AbstractNioSelector.java:349)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.(AbstractNioSelector.java:100)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.(AbstractNioWorker.java:52)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.(NioWorker.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorkerPool.createWorker(NioWorkerPool.java:45)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorkerPool.createWorker(NioWorkerPool.java:28)

Thanks,
Subhadip

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a124503b-ad60-42a0-bb17-de56c8c6a726%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Hi Ed,

I my case I have created a singleton TransportClient and querying ES with
the same in frequent interval to get data. I'm not doing any bulk
operations only searching index. threadPool needs to be shutdown once
Tomcat stops I guess, But when my webapplication is up and running is there
any need to shutdown the threadPool ? I'm getting this error recently.
Previously if the ES server goes down I used to get
NoNodeAvailableException and that's OK with me. But now the whole tomcat
goes down and no app is working showing OutOfMemoryError. Please suggest.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/00bad24d-b036-4613-b3f3-7a2802778d24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

It's hard to say what's truly causing it, but when your server goes down,
you should also see NoNodeAvailableException. Also, I don't think it
matters too much whether you are doing index or search requests, as all
requests queue handlers (Runnable) to the generic threadpool. What is your
query per second like when searching? What timeout have you set? I'm
wondering if TransportClient is flooding the threadpool with a whole bunch
of requests and causing your server to crash as a result. In this case
though, I might also expect to see timeout exceptions as well, but since
you haven't mentioned them, I'm not really sure.

For the threadpool shutdown, I typically include threadpool.shutdown()
whenever I call close() since my intention is to completely shutdown the
transportclient. I wouldn't suggest simply killing the threadpool only.

Having said that, have you tried protecting TransportClient from
handling/executing requests when NoNodeAvailableException occurs? We've set
up a wrapper class around TransportClient to queue up requests until
TransportClient recovers. We will periodically check to see if
TransportClient can communicate with ES, and then execute the queued
requests if things are back to normal. Worst case, we will drop the queued
ES requests and return an error message back to the client.

On Monday, January 12, 2015 at 4:38:09 AM UTC-8, Subhadip Bagui wrote:

Hi Ed,

I my case I have created a singleton TransportClient and querying ES with
the same in frequent interval to get data. I'm not doing any bulk
operations only searching index. threadPool needs to be shutdown once
Tomcat stops I guess, But when my webapplication is up and running is there
any need to shutdown the threadPool ? I'm getting this error recently.
Previously if the ES server goes down I used to get
NoNodeAvailableException and that's OK with me. But now the whole tomcat
goes down and no app is working showing OutOfMemoryError. Please suggest.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a01d7ebb-b86b-43f9-a93f-77c49b9be17f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.