# Memory leak related issue with Threadlocals in elasticsearch

**URL:** <https://discuss.elastic.co/t/memory-leak-related-issue-with-threadlocals-in-elasticsearch/20905>\
**Category:** Elasticsearch\
**Created:** [November 23, 2014, 3:36pm UTC](https://discuss.elastic.co/t/memory-leak-related-issue-with-threadlocals-in-elasticsearch/20905 "2014-11-23T15:36:23Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![prachicsa](https://avatars.discourse-cdn.com/v4/letter/p/f4b2a3/32.png) [@prachicsa](https://discuss.elastic.co/u/prachicsa)\
**Post date:** [November 23, 2014, 3:36pm UTC](https://discuss.elastic.co/t/memory-leak-related-issue-with-threadlocals-in-elasticsearch/20905/1 "2014-11-23T15:36:23Z")

</div>

I have a web service, where every time a button is pressed from UI, it  
connects to elastic search and fires a query. This is the code which is  
executed every time. The issue is that after a while, intermittently, the  
UI hangs.

private static final String CONFIG\_CLUSTER\_NAME = "cluster.name";

private static final String CLUSTER\_NAME = "sample\_es";  
private static final String[] transportAddress = {  
//Machine details  
};

private static final int transportPort = 9300;

public static Client initClient(){  
settings = ImmutableSettings.settingsBuilder().put(CONFIG\_CLUSTER\_NAME, CLUSTER\_NAME).build();

```
    Client client = new TransportClient(settings);
    for (int i=0 ; i < transportAddress.length-1 ; i++){
        ((TransportClient)client).addTransportAddress(new InetSocketTransportAddress(transportAddress[i], transportPort));
    }
    logger.info("TransportClient Created"); 
    return client;  

```

}

public static int query( String query) throws Exception  
{  
Client client = null;  
try{

```
        client = initClient();

        //Query search code

}catch(Exception e){
        e.printStackTrace();
}finally{
    if(client != null){
        client.close();
        logger.info("TransportClient Closed");
    }
}
    return result_count;

```

}

Whenever we restart the tomcat server, this is the error message we are  
seeing in logs. How should we fix this?

[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1] (value  
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1@5838ce3e]) and a value of type  
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom] (value  
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom@796c75b1]) but failed to remove it  
when the web application was stopped. This is very likely to create a memory leak.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [November 23, 2014, 4:18pm UTC](https://discuss.elastic.co/t/memory-leak-related-issue-with-threadlocals-in-elasticsearch/20905/2 "2014-11-23T16:18:20Z")

</div>

You do not correctly instantiate TransportClient as a singleton.

From the code snippet it is not clear, but it seems that you  
recreate TransportClient at each query, which is not correct.

Jörg

On Sun, Nov 23, 2014 at 4:36 PM, [prachicsa@gmail.com](mailto:prachicsa@gmail.com) wrote:

> I have a web service, where every time a button is pressed from UI, it  
> connects to Elasticsearch and fires a query. This is the code which is  
> executed every time. The issue is that after a while, intermittently, the  
> UI hangs.
> 
> private static final String CONFIG\_CLUSTER\_NAME = "cluster.name";
> 
> private static final String CLUSTER\_NAME = "sample\_es";  
> private static final String transportAddress = {  
> //Machine details  
> };
> 
> private static final int transportPort = 9300;
> 
> public static Client initClient(){  
> settings = ImmutableSettings.settingsBuilder().put(CONFIG\_CLUSTER\_NAME, CLUSTER\_NAME).build();
> 
> ```
> Client client = new TransportClient(settings);
> for (int i=0 ; i < transportAddress.length-1 ; i++){
> ((TransportClient)client).addTransportAddress(new InetSocketTransportAddress(transportAddress[i], transportPort));
> }
> logger.info("TransportClient Created");
> return client;
> 
> ```
> 
> }
> 
> public static int query( String query) throws Exception  
> {  
> Client client = null;  
> try{
> 
> ```
> client = initClient();
> 
> //Query search code
> 
> }catch(Exception e){
> e.printStackTrace();
> }finally{
> if(client != null){
> client.close();
> logger.info("TransportClient Closed");
> }
> }
> return result_count;
> 
> ```
> 
> }
> 
> Whenever we restart the tomcat server, this is the error message we are  
> seeing in logs. How should we fix this?
> 
> [org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1] (value  
> [org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1@5838ce3e]) and a value of type  
> [org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom] (value  
> [org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom@796c75b1]) but failed to remove it  
> when the web application was stopped. This is very likely to create a memory leak.
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFP4nRc9du3HZV\_%3DQgXJDCg4jV2V3oT%2B%3DQ0rsK3\_uvmyg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFP4nRc9du3HZV_%3DQgXJDCg4jV2V3oT%2B%3DQ0rsK3_uvmyg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![prachicsa](https://avatars.discourse-cdn.com/v4/letter/p/f4b2a3/32.png) [@prachicsa](https://discuss.elastic.co/u/prachicsa)\
**Post date:** [November 24, 2014, 9:35am UTC](https://discuss.elastic.co/t/memory-leak-related-issue-with-threadlocals-in-elasticsearch/20905/3 "2014-11-24T09:35:03Z")

</div>

is it happening because we are creating and closing a new client for every  
query??

> <https://github.com/elastic/elasticsearch/issues/283>
>
> I am running ElasticSearch 0.9.0. My web application uses the Java API to connec…t to the server. After closing the client node, several API objects are still referenced by ThreadLocals. As a consequence the web application can not be garbage collected after being undeployed. This creates a memory leak.
> 
> \`\`\`
> 30.07.2010 14:59:07 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
> SCHWERWIEGEND: The web application \[\] created a ThreadLocal with key of type \[null\] (value \[org.elasticsearch.common.inject.InjectorImpl$1@19d44df\]) and a value of type \[java.lang.Object\[\]\] (value \[\[Ljava.lang.Object;@149895\]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
> 30.07.2010 14:59:07 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
> SCHWERWIEGEND: The web application \[\] created a ThreadLocal with key of type \[null\] (value \[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1@2d8a59\]) and a value of type \[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom\] (value \[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom@1d60b6a\]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
> 30.07.2010 14:59:07 org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
> SCHWERWIEGEND: The web application \[\] created a ThreadLocal with key of type \[null\] (value \[org.elasticsearch.common.inject.InjectorImpl$1@19d44df\]) and a value of type \[java.lang.Object\[\]\] (value \[\[Ljava.lang.Object;@1876a6d\]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
> \`\`\`

Will having only one connection forever solve this? Does ES connection stay  
live even if no one uses it for a couple hours? Or it becomes stale?

That is why I am creating and closing a connection everytime.

On Sunday, November 23, 2014 9:48:49 PM UTC+5:30, Jörg Prante wrote:

> You do not correctly instantiate TransportClient as a singleton.
> 
> From the code snippet it is not clear, but it seems that you  
> recreate TransportClient at each query, which is not correct.
> 
> Jörg
> 
> On Sun, Nov 23, 2014 at 4:36 PM, \<[prac...@gmail.com](mailto:prac...@gmail.com) \<javascript:\>\> wrote:
> 
> > I have a web service, where every time a button is pressed from UI, it  
> > connects to Elasticsearch and fires a query. This is the code which is  
> > executed every time. The issue is that after a while, intermittently, the  
> > UI hangs.
> > 
> > private static final String CONFIG\_CLUSTER\_NAME = "cluster.name";
> > 
> > private static final String CLUSTER\_NAME = "sample\_es";  
> > private static final String transportAddress = {  
> > //Machine details  
> > };
> > 
> > private static final int transportPort = 9300;
> > 
> > public static Client initClient(){  
> > settings = ImmutableSettings.settingsBuilder().put(CONFIG\_CLUSTER\_NAME, CLUSTER\_NAME).build();
> > 
> > ```
> > Client client = new TransportClient(settings);
> > for (int i=0 ; i < transportAddress.length-1 ; i++){
> > ((TransportClient)client).addTransportAddress(new InetSocketTransportAddress(transportAddress[i], transportPort));
> > }
> > logger.info("TransportClient Created"); 
> > return client;  
> > 
> > ```
> > 
> > }
> > 
> > public static int query( String query) throws Exception  
> > {  
> > Client client = null;  
> > try{
> > 
> > ```
> > client = initClient();
> > 
> > //Query search code
> > 
> > }catch(Exception e){
> > e.printStackTrace();
> > }finally{
> > if(client != null){
> > client.close();
> > logger.info("TransportClient Closed");
> > }
> > }
> > return result_count;
> > 
> > ```
> > 
> > }
> > 
> > Whenever we restart the tomcat server, this is the error message we are  
> > seeing in logs. How should we fix this?
> > 
> > [org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1] (value  
> > [org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1@5838ce3e]) and a value of type  
> > [org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom] (value  
> > [org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom@796c75b1]) but failed to remove it  
> > when the web application was stopped. This is very likely to create a memory leak.
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/101a0f86-4fc5-415c-957a-71cf21b14f5f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/101a0f86-4fc5-415c-957a-71cf21b14f5f%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [November 24, 2014, 9:48am UTC](https://discuss.elastic.co/t/memory-leak-related-issue-with-threadlocals-in-elasticsearch/20905/4 "2014-11-24T09:48:39Z")

</div>

TransportClient uses a thread pool with own management and opens  
connections on demand. The TransportClient is a singleton container for  
this.

Creating and closing TransportClient is like restarting Tomcat just because  
you have just to process a new request for a web app.

The Tomcat memory leak warnings are because Tomcat thinks he has to manage  
TransportClient thread pools and resources.

Jörg

On Mon, Nov 24, 2014 at 10:35 AM, [prachicsa@gmail.com](mailto:prachicsa@gmail.com) wrote:

> is it happening because we are creating and closing a new client for every  
> query??
> 
> [ThreadLocals not cleared · Issue #283 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/283)
> 
> Will having only one connection forever solve this? Does ES connection  
> stay live even if no one uses it for a couple hours? Or it becomes stale?
> 
> That is why I am creating and closing a connection everytime.
> 
> On Sunday, November 23, 2014 9:48:49 PM UTC+5:30, Jörg Prante wrote:
> 
> > You do not correctly instantiate TransportClient as a singleton.
> > 
> > From the code snippet it is not clear, but it seems that you  
> > recreate TransportClient at each query, which is not correct.
> > 
> > Jörg
> > 
> > On Sun, Nov 23, 2014 at 4:36 PM, [prac...@gmail.com](mailto:prac...@gmail.com) wrote:
> > 
> > > I have a web service, where every time a button is pressed from UI, it  
> > > connects to Elasticsearch and fires a query. This is the code which is  
> > > executed every time. The issue is that after a while, intermittently, the  
> > > UI hangs.
> > > 
> > > private static final String CONFIG\_CLUSTER\_NAME = "cluster.name";
> > > 
> > > private static final String CLUSTER\_NAME = "sample\_es";  
> > > private static final String transportAddress = {  
> > > //Machine details  
> > > };
> > > 
> > > private static final int transportPort = 9300;
> > > 
> > > public static Client initClient(){  
> > > settings = ImmutableSettings.settingsBuilder().put(CONFIG\_CLUSTER\_NAME, CLUSTER\_NAME).build();
> > > 
> > > ```
> > > Client client = new TransportClient(settings);
> > > for (int i=0 ; i < transportAddress.length-1 ; i++){
> > > ((TransportClient)client).addTransportAddress(new InetSocketTransportAddress(transportAddress[i], transportPort));
> > > }
> > > logger.info("TransportClient Created");
> > > return client;
> > > 
> > > ```
> > > 
> > > }
> > > 
> > > public static int query( String query) throws Exception  
> > > {  
> > > Client client = null;  
> > > try{
> > > 
> > > ```
> > > client = initClient();
> > > 
> > > //Query search code
> > > 
> > > }catch(Exception e){
> > > e.printStackTrace();
> > > }finally{
> > > if(client != null){
> > > client.close();
> > > logger.info("TransportClient Closed");
> > > }
> > > }
> > > return result_count;
> > > 
> > > ```
> > > 
> > > }
> > > 
> > > Whenever we restart the tomcat server, this is the error message we are  
> > > seeing in logs. How should we fix this?
> > > 
> > > [org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1] (value  
> > > [org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1@5838ce3e]) and a value of type  
> > > [org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom] (value  
> > > [org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom@796c75b1]) but failed to remove it  
> > > when the web application was stopped. This is very likely to create a memory leak.
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)  
> > > msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%  
> > > [40googlegroups.com](http://40googlegroups.com)  
> > > [https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .  
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/101a0f86-4fc5-415c-957a-71cf21b14f5f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/101a0f86-4fc5-415c-957a-71cf21b14f5f%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/101a0f86-4fc5-415c-957a-71cf21b14f5f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/101a0f86-4fc5-415c-957a-71cf21b14f5f%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFi-oyoHuY4jMvwMhav0Eoc4AHMeziXga0J7xg2ypH%3DJg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFi-oyoHuY4jMvwMhav0Eoc4AHMeziXga0J7xg2ypH%3DJg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:48am UTC](https://discuss.elastic.co/t/memory-leak-related-issue-with-threadlocals-in-elasticsearch/20905/5 "2017-07-06T00:48:08Z")

</div>