Memory leak related issue with Threadlocals in elasticsearch

Elastic Stack Elasticsearch
1

I have a web service, where every time a button is pressed from UI, it
connects to elastic search and fires a query. This is the code which is
executed every time. The issue is that after a while, intermittently, the
UI hangs.

private static final String CONFIG_CLUSTER_NAME = "cluster.name";

private static final String CLUSTER_NAME = "sample_es";
private static final String[] transportAddress = {
//Machine details
};

private static final int transportPort = 9300;

public static Client initClient(){
settings = ImmutableSettings.settingsBuilder().put(CONFIG_CLUSTER_NAME, CLUSTER_NAME).build();

    Client client = new TransportClient(settings);
    for (int i=0 ; i < transportAddress.length-1 ; i++){
        ((TransportClient)client).addTransportAddress(new InetSocketTransportAddress(transportAddress[i], transportPort));
    }
    logger.info("TransportClient Created"); 
    return client;

}

public static int query( String query) throws Exception
{
Client client = null;
try{

        client = initClient();

        //Query search code

}catch(Exception e){
        e.printStackTrace();
}finally{
    if(client != null){
        client.close();
        logger.info("TransportClient Closed");
    }
}
    return result_count;

}

Whenever we restart the tomcat server, this is the error message we are
seeing in logs. How should we fix this?

[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1] (value
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1@5838ce3e]) and a value of type
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom] (value
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom@796c75b1]) but failed to remove it
when the web application was stopped. This is very likely to create a memory leak.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

You do not correctly instantiate TransportClient as a singleton.

From the code snippet it is not clear, but it seems that you
recreate TransportClient at each query, which is not correct.

Jörg

On Sun, Nov 23, 2014 at 4:36 PM, prachicsa@gmail.com wrote:

I have a web service, where every time a button is pressed from UI, it
connects to Elasticsearch and fires a query. This is the code which is
executed every time. The issue is that after a while, intermittently, the
UI hangs.

private static final String CONFIG_CLUSTER_NAME = "cluster.name";

private static final String CLUSTER_NAME = "sample_es";
private static final String transportAddress = {
//Machine details
};

private static final int transportPort = 9300;

public static Client initClient(){
settings = ImmutableSettings.settingsBuilder().put(CONFIG_CLUSTER_NAME, CLUSTER_NAME).build();

    Client client = new TransportClient(settings);
    for (int i=0 ; i < transportAddress.length-1 ; i++){
        ((TransportClient)client).addTransportAddress(new InetSocketTransportAddress(transportAddress[i], transportPort));
    }
    logger.info("TransportClient Created");
    return client;

}

public static int query( String query) throws Exception
{
Client client = null;
try{

        client = initClient();

        //Query search code

}catch(Exception e){
        e.printStackTrace();
}finally{
    if(client != null){
        client.close();
        logger.info("TransportClient Closed");
    }
}
    return result_count;

}

Whenever we restart the tomcat server, this is the error message we are
seeing in logs. How should we fix this?

[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1] (value
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1@5838ce3e]) and a value of type
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom] (value
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom@796c75b1]) but failed to remove it
when the web application was stopped. This is very likely to create a memory leak.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFP4nRc9du3HZV_%3DQgXJDCg4jV2V3oT%2B%3DQ0rsK3_uvmyg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

is it happening because we are creating and closing a new client for every
query??

Will having only one connection forever solve this? Does ES connection stay
live even if no one uses it for a couple hours? Or it becomes stale?

That is why I am creating and closing a connection everytime.

On Sunday, November 23, 2014 9:48:49 PM UTC+5:30, Jörg Prante wrote:

You do not correctly instantiate TransportClient as a singleton.

From the code snippet it is not clear, but it seems that you
recreate TransportClient at each query, which is not correct.

Jörg

On Sun, Nov 23, 2014 at 4:36 PM, <prac...@gmail.com <javascript:>> wrote:

I have a web service, where every time a button is pressed from UI, it
connects to Elasticsearch and fires a query. This is the code which is
executed every time. The issue is that after a while, intermittently, the
UI hangs.

private static final String CONFIG_CLUSTER_NAME = "cluster.name";

private static final String CLUSTER_NAME = "sample_es";
private static final String transportAddress = {
//Machine details
};

private static final int transportPort = 9300;

public static Client initClient(){
settings = ImmutableSettings.settingsBuilder().put(CONFIG_CLUSTER_NAME, CLUSTER_NAME).build();

    Client client = new TransportClient(settings);
    for (int i=0 ; i < transportAddress.length-1 ; i++){
        ((TransportClient)client).addTransportAddress(new InetSocketTransportAddress(transportAddress[i], transportPort));
    }
    logger.info("TransportClient Created"); 
    return client;

}

public static int query( String query) throws Exception
{
Client client = null;
try{

        client = initClient();

        //Query search code

}catch(Exception e){
        e.printStackTrace();
}finally{
    if(client != null){
        client.close();
        logger.info("TransportClient Closed");
    }
}
    return result_count;

}

Whenever we restart the tomcat server, this is the error message we are
seeing in logs. How should we fix this?

[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1] (value
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1@5838ce3e]) and a value of type
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom] (value
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom@796c75b1]) but failed to remove it
when the web application was stopped. This is very likely to create a memory leak.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/101a0f86-4fc5-415c-957a-71cf21b14f5f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

TransportClient uses a thread pool with own management and opens
connections on demand. The TransportClient is a singleton container for
this.

Creating and closing TransportClient is like restarting Tomcat just because
you have just to process a new request for a web app.

The Tomcat memory leak warnings are because Tomcat thinks he has to manage
TransportClient thread pools and resources.

Jörg

On Mon, Nov 24, 2014 at 10:35 AM, prachicsa@gmail.com wrote:

is it happening because we are creating and closing a new client for every
query??

ThreadLocals not cleared · Issue #283 · elastic/elasticsearch · GitHub

Will having only one connection forever solve this? Does ES connection
stay live even if no one uses it for a couple hours? Or it becomes stale?

That is why I am creating and closing a connection everytime.

On Sunday, November 23, 2014 9:48:49 PM UTC+5:30, Jörg Prante wrote:

You do not correctly instantiate TransportClient as a singleton.

From the code snippet it is not clear, but it seems that you
recreate TransportClient at each query, which is not correct.

Jörg

On Sun, Nov 23, 2014 at 4:36 PM, prac...@gmail.com wrote:

I have a web service, where every time a button is pressed from UI, it
connects to Elasticsearch and fires a query. This is the code which is
executed every time. The issue is that after a while, intermittently, the
UI hangs.

private static final String CONFIG_CLUSTER_NAME = "cluster.name";

private static final String CLUSTER_NAME = "sample_es";
private static final String transportAddress = {
//Machine details
};

private static final int transportPort = 9300;

public static Client initClient(){
settings = ImmutableSettings.settingsBuilder().put(CONFIG_CLUSTER_NAME, CLUSTER_NAME).build();

    Client client = new TransportClient(settings);
    for (int i=0 ; i < transportAddress.length-1 ; i++){
        ((TransportClient)client).addTransportAddress(new InetSocketTransportAddress(transportAddress[i], transportPort));
    }
    logger.info("TransportClient Created");
    return client;

}

public static int query( String query) throws Exception
{
Client client = null;
try{

        client = initClient();

        //Query search code

}catch(Exception e){
        e.printStackTrace();
}finally{
    if(client != null){
        client.close();
        logger.info("TransportClient Closed");
    }
}
    return result_count;

}

Whenever we restart the tomcat server, this is the error message we are
seeing in logs. How should we fix this?

[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1] (value
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom$1@5838ce3e]) and a value of type
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom] (value
[org.elasticsearch.common.util.concurrent.jsr166y.ThreadLocalRandom@796c75b1]) but failed to remove it
when the web application was stopped. This is very likely to create a memory leak.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/58af775d-5db3-42ec-bdec-0a8d13f9d76b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/101a0f86-4fc5-415c-957a-71cf21b14f5f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/101a0f86-4fc5-415c-957a-71cf21b14f5f%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFi-oyoHuY4jMvwMhav0Eoc4AHMeziXga0J7xg2ypH%3DJg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.