# Elasticsearch client is giving OutOfMemoryError once connection is lost to elaticsearch server
**URL:** https://discuss.elastic.co/t/elasticsearch-client-is-giving-outofmemoryerror-once-connection-is-lost-to-elaticsearch-server/21570
**Category:** Elasticsearch
**Created:** [January 10, 2015, 5:29pm UTC](https://discuss.elastic.co/t/elasticsearch-client-is-giving-outofmemoryerror-once-connection-is-lost-to-elaticsearch-server/21570 "2015-01-10T17:29:56Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@bagui](https://discuss.elastic.co/u/bagui)
#### Post date: [January 10, 2015, 5:29pm UTC](https://discuss.elastic.co/t/elasticsearch-client-is-giving-outofmemoryerror-once-connection-is-lost-to-elaticsearch-server/21570/1 "2015-01-10T17:29:56Z")
Hi,
I'm using elasticsearch using TransportClient for multiple operation. The
issue I'm facing now is if my es server goes down my client side app
getting OutOfMemoryError. Getting the below exception. I had to restart my
tomcat every time after this to make my application up. Can some one please
suggest how to prevent this.
Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [spring] in context with path
[/aricloud] threw exception [Handler processing failed; nested exception is
java.lang.OutOfMemoryError: unable to create new native thread] with root
cause
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:349)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:100)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:52)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
Thanks,
Subhadip
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6930af3b-13ad-476c-82cc-7c8d792dea67%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6930af3b-13ad-476c-82cc-7c8d792dea67%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Ed\_Kim](https://discuss.elastic.co/u/Ed_Kim)
#### Post date: [January 11, 2015, 7:03am UTC](https://discuss.elastic.co/t/elasticsearch-client-is-giving-outofmemoryerror-once-connection-is-lost-to-elaticsearch-server/21570/2 "2015-01-11T07:03:32Z")
Other members can correct me if I'm wrong, but I notice that when you lose
connection with the server, the transportclient queues retries of whatever
operations you try to execute, and it starts to queue listeners into a
'generic' threadpool (which I read somewhere that it was unbounded). We've
seen this problem when we thrash ES until it eventually stops responding,
and our bulk requests start to back up and eventually cause the application
to halt due to OOM.
I don't know exactly what your application is doing when your ES node(s) go
down, but perhaps you can proactively stop requests to ES servers once your
application sees the no node exception error (which you should get when ES
goes down). You could also close the transportclient and shutdown its
threadpool and reconnect/instantiate after a timed delay to clean up
whatever is floating around in the transportclient. We have been able to
solve most of our native thread issues by protecting our use of
transportclient and doing a soft restart of this client.
On Saturday, January 10, 2015 at 9:29:56 AM UTC-8, Subhadip Bagui wrote:
> Hi,
>
> I'm using elasticsearch using TransportClient for multiple operation. The
> issue I'm facing now is if my es server goes down my client side app
> getting OutOfMemoryError. Getting the below exception. I had to restart my
> tomcat every time after this to make my application up. Can some one please
> suggest how to prevent this.
>
> Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve invoke
> SEVERE: Servlet.service() for servlet [spring] in context with path
> [/aricloud] threw exception [Handler processing failed; nested exception is
> java.lang.OutOfMemoryError: unable to create new native thread] with root
> cause
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:640)
> at
> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
> at
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
> at
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:349)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:100)
> at
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:52)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.(NioWorker.java:45)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45)
> at
> org.elasticsearch.common.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28)
>
> Thanks,
> Subhadip
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)
#### Post date: [January 12, 2015, 3:22am UTC](https://discuss.elastic.co/t/elasticsearch-client-is-giving-outofmemoryerror-once-connection-is-lost-to-elaticsearch-server/21570/3 "2015-01-12T03:22:14Z")
Hi Ed,
soft restart as in client.close() ? I did that as well, does not seem to
work, hence I resolve to tomcat restart instead...
Jason
On Sun, Jan 11, 2015 at 3:03 PM, Ed Kim [edkim81@gmail.com](mailto:edkim81@gmail.com) wrote:
> Other members can correct me if I'm wrong, but I notice that when you lose
> connection with the server, the transportclient queues retries of whatever
> operations you try to execute, and it starts to queue listeners into a
> 'generic' threadpool (which I read somewhere that it was unbounded). We've
> seen this problem when we thrash ES until it eventually stops responding,
> and our bulk requests start to back up and eventually cause the application
> to halt due to OOM.
>
> I don't know exactly what your application is doing when your ES node(s)
> go down, but perhaps you can proactively stop requests to ES servers once
> your application sees the no node exception error (which you should get
> when ES goes down). You could also close the transportclient and shutdown
> its threadpool and reconnect/instantiate after a timed delay to clean up
> whatever is floating around in the transportclient. We have been able to
> solve most of our native thread issues by protecting our use of
> transportclient and doing a soft restart of this client.
>
> On Saturday, January 10, 2015 at 9:29:56 AM UTC-8, Subhadip Bagui wrote:
>
> > Hi,
> >
> > I'm using elasticsearch using TransportClient for multiple operation. The
> > issue I'm facing now is if my es server goes down my client side app
> > getting OutOfMemoryError. Getting the below exception. I had to restart my
> > tomcat every time after this to make my application up. Can some one please
> > suggest how to prevent this.
> >
> > Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve
> > invoke
> > SEVERE: Servlet.service() for servlet [spring] in context with path
> > [/aricloud] threw exception [Handler processing failed; nested exception is
> > java.lang.OutOfMemoryError: unable to create new native thread] with root
> > cause
> > java.lang.OutOfMemoryError: unable to create new native thread
> > at java.lang.Thread.start0(Native Method)
> > at java.lang.Thread.start(Thread.java:640)
> > at java.util.concurrent.ThreadPoolExecutor.addThread(
> > ThreadPoolExecutor.java:681)
> > at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(
> > ThreadPoolExecutor.java:727)
> > at java.util.concurrent.ThreadPoolExecutor.execute(
> > ThreadPoolExecutor.java:655)
> > at org.elasticsearch.common.netty.util.internal.
> > DeadLockProofWorker.start(DeadLockProofWorker.java:38)
> > at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.
> > openSelector(AbstractNioSelector.java:349)
> > at org.elasticsearch.common.netty.channel.socket.nio.
> > AbstractNioSelector.(AbstractNioSelector.java:100)
> > at org.elasticsearch.common.netty.channel.socket.nio.
> > AbstractNioWorker.(AbstractNioWorker.java:52)
> > at org.elasticsearch.common.netty.channel.socket.nio.
> > NioWorker.(NioWorker.java:45)
> > at org.elasticsearch.common.netty.channel.socket.nio.
> > NioWorkerPool.createWorker(NioWorkerPool.java:45)
> > at org.elasticsearch.common.netty.channel.socket.nio.
> > NioWorkerPool.createWorker(NioWorkerPool.java:28)
> >
> > Thanks,
> > Subhadip
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4ityTv%2B2-tQLrqWQetKfL0juxDVqhsJeNbYQkZbzr6kjtDg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4ityTv%2B2-tQLrqWQetKfL0juxDVqhsJeNbYQkZbzr6kjtDg%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Ed\_Kim](https://discuss.elastic.co/u/Ed_Kim)
#### Post date: [January 12, 2015, 7:40am UTC](https://discuss.elastic.co/t/elasticsearch-client-is-giving-outofmemoryerror-once-connection-is-lost-to-elaticsearch-server/21570/4 "2015-01-12T07:40:47Z")
Yes, but in addition I would also call client.threadPool().shutdown() to
make sure you free up the threadpool.
On Sunday, January 11, 2015 at 7:22:20 PM UTC-8, Jason Wee wrote:
> Hi Ed,
>
> soft restart as in client.close() ? I did that as well, does not seem to
> work, hence I resolve to tomcat restart instead...
>
> Jason
>
> On Sun, Jan 11, 2015 at 3:03 PM, Ed Kim \<[edk...@gmail.com](mailto:edk...@gmail.com) \\>
> wrote:
>
> > Other members can correct me if I'm wrong, but I notice that when you
> > lose connection with the server, the transportclient queues retries of
> > whatever operations you try to execute, and it starts to queue listeners
> > into a 'generic' threadpool (which I read somewhere that it was unbounded).
> > We've seen this problem when we thrash ES until it eventually stops
> > responding, and our bulk requests start to back up and eventually cause the
> > application to halt due to OOM.
> >
> > I don't know exactly what your application is doing when your ES node(s)
> > go down, but perhaps you can proactively stop requests to ES servers once
> > your application sees the no node exception error (which you should get
> > when ES goes down). You could also close the transportclient and shutdown
> > its threadpool and reconnect/instantiate after a timed delay to clean up
> > whatever is floating around in the transportclient. We have been able to
> > solve most of our native thread issues by protecting our use of
> > transportclient and doing a soft restart of this client.
> >
> > On Saturday, January 10, 2015 at 9:29:56 AM UTC-8, Subhadip Bagui wrote:
> >
> > > Hi,
> > >
> > > I'm using elasticsearch using TransportClient for multiple operation.
> > > The issue I'm facing now is if my es server goes down my client side app
> > > getting OutOfMemoryError. Getting the below exception. I had to restart my
> > > tomcat every time after this to make my application up. Can some one please
> > > suggest how to prevent this.
> > >
> > > Jan 9, 2015 5:38:44 PM org.apache.catalina.core.StandardWrapperValve
> > > invoke
> > > SEVERE: Servlet.service() for servlet [spring] in context with path
> > > [/aricloud] threw exception [Handler processing failed; nested exception is
> > > java.lang.OutOfMemoryError: unable to create new native thread] with root
> > > cause
> > > java.lang.OutOfMemoryError: unable to create new native thread
> > > at java.lang.Thread.start0(Native Method)
> > > at java.lang.Thread.start(Thread.java:640)
> > > at java.util.concurrent.ThreadPoolExecutor.addThread(
> > > ThreadPoolExecutor.java:681)
> > > at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(
> > > ThreadPoolExecutor.java:727)
> > > at java.util.concurrent.ThreadPoolExecutor.execute(
> > > ThreadPoolExecutor.java:655)
> > > at org.elasticsearch.common.netty.util.internal.
> > > DeadLockProofWorker.start(DeadLockProofWorker.java:38)
> > > at org.elasticsearch.common.netty.channel.socket.nio.
> > > AbstractNioSelector.openSelector(AbstractNioSelector.java:349)
> > > at org.elasticsearch.common.netty.channel.socket.nio.
> > > AbstractNioSelector.(AbstractNioSelector.java:100)
> > > at org.elasticsearch.common.netty.channel.socket.nio.
> > > AbstractNioWorker.(AbstractNioWorker.java:52)
> > > at org.elasticsearch.common.netty.channel.socket.nio.
> > > NioWorker.(NioWorker.java:45)
> > > at org.elasticsearch.common.netty.channel.socket.nio.
> > > NioWorkerPool.createWorker(NioWorkerPool.java:45)
> > > at org.elasticsearch.common.netty.channel.socket.nio.
> > > NioWorkerPool.createWorker(NioWorkerPool.java:28)
> > >
> > > Thanks,
> > > Subhadip
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0bbbbab9-8356-4ca5-b53c-b682cbd76b1a%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a124503b-ad60-42a0-bb17-de56c8c6a726%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a124503b-ad60-42a0-bb17-de56c8c6a726%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@bagui](https://discuss.elastic.co/u/bagui)
#### Post date: [January 12, 2015, 12:38pm UTC](https://discuss.elastic.co/t/elasticsearch-client-is-giving-outofmemoryerror-once-connection-is-lost-to-elaticsearch-server/21570/5 "2015-01-12T12:38:09Z")
Hi Ed,
I my case I have created a singleton TransportClient and querying ES with
the same in frequent interval to get data. I'm not doing any bulk
operations only searching index. threadPool needs to be shutdown once
Tomcat stops I guess, But when my webapplication is up and running is there
any need to shutdown the threadPool ? I'm getting this error recently.
Previously if the ES server goes down I used to get
NoNodeAvailableException and that's OK with me. But now the whole tomcat
goes down and no app is working showing OutOfMemoryError. Please suggest.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/00bad24d-b036-4613-b3f3-7a2802778d24%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/00bad24d-b036-4613-b3f3-7a2802778d24%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Ed\_Kim](https://discuss.elastic.co/u/Ed_Kim)
#### Post date: [January 12, 2015, 7:55pm UTC](https://discuss.elastic.co/t/elasticsearch-client-is-giving-outofmemoryerror-once-connection-is-lost-to-elaticsearch-server/21570/6 "2015-01-12T19:55:29Z")
It's hard to say what's truly causing it, but when your server goes down,
you should also see NoNodeAvailableException. Also, I don't think it
matters too much whether you are doing index or search requests, as all
requests queue handlers (Runnable) to the generic threadpool. What is your
query per second like when searching? What timeout have you set? I'm
wondering if TransportClient is flooding the threadpool with a whole bunch
of requests and causing your server to crash as a result. In this case
though, I might also expect to see timeout exceptions as well, but since
you haven't mentioned them, I'm not really sure.
For the threadpool shutdown, I typically include threadpool.shutdown()
whenever I call close() since my intention is to completely shutdown the
transportclient. I wouldn't suggest simply killing the threadpool only.
Having said that, have you tried protecting TransportClient from
handling/executing requests when NoNodeAvailableException occurs? We've set
up a wrapper class around TransportClient to queue up requests until
TransportClient recovers. We will periodically check to see if
TransportClient can communicate with ES, and then execute the queued
requests if things are back to normal. Worst case, we will drop the queued
ES requests and return an error message back to the client.
On Monday, January 12, 2015 at 4:38:09 AM UTC-8, Subhadip Bagui wrote:
> Hi Ed,
>
> I my case I have created a singleton TransportClient and querying ES with
> the same in frequent interval to get data. I'm not doing any bulk
> operations only searching index. threadPool needs to be shutdown once
> Tomcat stops I guess, But when my webapplication is up and running is there
> any need to shutdown the threadPool ? I'm getting this error recently.
> Previously if the ES server goes down I used to get
> NoNodeAvailableException and that's OK with me. But now the whole tomcat
> goes down and no app is working showing OutOfMemoryError. Please suggest.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a01d7ebb-b86b-43f9-a93f-77c49b9be17f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a01d7ebb-b86b-43f9-a93f-77c49b9be17f%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:39am UTC](https://discuss.elastic.co/t/elasticsearch-client-is-giving-outofmemoryerror-once-connection-is-lost-to-elaticsearch-server/21570/7 "2017-07-06T00:39:28Z")