Java TransportClient Memory Leak

kelaban · March 14, 2013, 1:57pm

I've had an ongoing problem using the Java TransportClient injected through
elasticsearch-spring. When trying to load test my client once I hit the
180k request mark requests start to hang, some take 20+ seconds. I've load
tested the elasticsearch directly to make sure it wasn't a configuration
problem over there. When I kill my load test if I start it up again it
picks up just as slow until I restart my client. After checking the GC logs
fingers point to memory being used up.
This is how I am doing my search:

    String escapedQuery = LUCENE_PATTERN.matcher(params.getQuery()).

replaceAll(REPLACEMENT_STRING);

    SearchRequestBuilder theSearch = esClient.prepareSearch(THE_INDEX)
            .setSearchType(SearchType.QUERY_THEN_FETCH);


    theSearch.addFields(params.getAsStringArray(params.getReturnFields

()));

    QueryBuilder theQuery = QueryBuilders.queryString(escapedQuery)
            .field(TICKER_FIELD)
            .field(NAME_FIELD)
            .analyzer(WHITESPACE_ANALYZER)
            .useDisMax(true)
            .defaultOperator(QueryStringQueryBuilder.Operator.AND);


    theSearch.setQuery(theQuery)
            .setSize(params.getSize())
            .setFrom(params.getFrom());


    SearchResponse resp = theSearch.execute().actionGet();

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

kelaban · March 14, 2013, 1:58pm

To clarify I am using ES v0.20.5

On Thursday, March 14, 2013 9:57:45 AM UTC-4, Keith L wrote:

I've had an ongoing problem using the Java TransportClient injected
through elasticsearch-spring. When trying to load test my client once I hit
the 180k request mark requests start to hang, some take 20+ seconds. I've
load tested the elasticsearch directly to make sure it wasn't a
configuration problem over there. When I kill my load test if I start it up
again it picks up just as slow until I restart my client. After checking
the GC logs fingers point to memory being used up.
This is how I am doing my search:
    String escapedQuery = LUCENE_PATTERN.matcher(params.getQuery()).
replaceAll(REPLACEMENT_STRING);
    SearchRequestBuilder theSearch = esClient.prepareSearch(THE_INDEX)
            .setSearchType(SearchType.QUERY_THEN_FETCH);


    theSearch.addFields(params.getAsStringArray(params.getReturnFields
()));
    QueryBuilder theQuery = QueryBuilders.queryString(escapedQuery)
            .field(TICKER_FIELD)
            .field(NAME_FIELD)
            .analyzer(WHITESPACE_ANALYZER)
            .useDisMax(true)
            .defaultOperator(QueryStringQueryBuilder.Operator.AND);


    theSearch.setQuery(theQuery)
            .setSize(params.getSize())
            .setFrom(params.getFrom());


    SearchResponse resp = theSearch.execute().actionGet();

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

kelaban · March 14, 2013, 1:59pm

to clarify I am using ES v0.20.5

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

kelaban · March 14, 2013, 2:02pm

to clarify I am using ES v0.20.5. And this is in a tomcat webapp

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · March 14, 2013, 2:22pm

Are you restarting your webapp (modify code live) instead of restarting Tomcat?

If so, it's due to a very old issue in Google guice as far as I remember. Some threads are not deleted and still here in the JVM when restarting the webapp.

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 14 mars 2013 à 15:02, Keith L kelaban17@gmail.com a écrit :

to clarify I am using ES v0.20.5. And this is in a tomcat webapp

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

kelaban · March 14, 2013, 2:33pm

I think that's not the issue. When I restart the tomcat everything works as
expected again until I get to the 180k request mark at which point it
cannot handle anymore requests until the app is restarted again. The app
conks out at 180k almost on the mark every time regardless of stress
testing with 1 thread or 100 threads and always preforms the the same after
the webapp is restarted.

On Thursday, March 14, 2013 10:22:44 AM UTC-4, David Pilato wrote:

Are you restarting your webapp (modify code live) instead of restarting
Tomcat?

If so, it's due to a very old issue in Google guice as far as I remember.
Some threads are not deleted and still here in the JVM when restarting the
webapp.

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 14 mars 2013 à 15:02, Keith L <kela...@gmail.com <javascript:>> a
écrit :

to clarify I am using ES v0.20.5. And this is in a tomcat webapp

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

kelaban · March 15, 2013, 5:08pm

Figured out the issue, turns out not to be elasticsearch. The default
tomcat session is set to 30 minutes which was taking up all the memory.
After fixing that issue the elasticsearch load test performed perfect.

On Thursday, March 14, 2013 10:33:17 AM UTC-4, Keith L wrote:

I think that's not the issue. When I restart the tomcat everything works
as expected again until I get to the 180k request mark at which point it
cannot handle anymore requests until the app is restarted again. The app
conks out at 180k almost on the mark every time regardless of stress
testing with 1 thread or 100 threads and always preforms the the same after
the webapp is restarted.

On Thursday, March 14, 2013 10:22:44 AM UTC-4, David Pilato wrote:

Are you restarting your webapp (modify code live) instead of restarting
Tomcat?

If so, it's due to a very old issue in Google guice as far as I remember.
Some threads are not deleted and still here in the JVM when restarting the
webapp.

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 14 mars 2013 à 15:02, Keith L kela...@gmail.com a écrit :

to clarify I am using ES v0.20.5. And this is in a tomcat webapp

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

dadoonet · March 15, 2013, 5:14pm

Thanks for the update.

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 15 mars 2013 à 18:08, Keith L kelaban17@gmail.com a écrit :

Figured out the issue, turns out not to be elasticsearch. The default tomcat session is set to 30 minutes which was taking up all the memory. After fixing that issue the elasticsearch load test performed perfect.

On Thursday, March 14, 2013 10:33:17 AM UTC-4, Keith L wrote:

I think that's not the issue. When I restart the tomcat everything works as expected again until I get to the 180k request mark at which point it cannot handle anymore requests until the app is restarted again. The app conks out at 180k almost on the mark every time regardless of stress testing with 1 thread or 100 threads and always preforms the the same after the webapp is restarted.

On Thursday, March 14, 2013 10:22:44 AM UTC-4, David Pilato wrote:

Are you restarting your webapp (modify code live) instead of restarting Tomcat?

If so, it's due to a very old issue in Google guice as far as I remember. Some threads are not deleted and still here in the JVM when restarting the webapp.

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 14 mars 2013 à 15:02, Keith L kela...@gmail.com a écrit :

to clarify I am using ES v0.20.5. And this is in a tomcat webapp

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.