Tomcat / webapp with ES embedded in eclipse (debug mode)

dadoonet · January 4, 2012, 8:50pm

Hi there,

I'm playing with an embedded ES node in a webapp deployed in a tomcat
container.

ES Node is injected using Spring.

ES is 0.18.5

Tomcat is 7.0.19

Spring is 3.1.0.RELEASE

And I'm launching everything in debug mode from Eclipse.

Everything works fine for the "first launch" but each time tomcat restarts
(because something changes in my workspace), it seems that Elasticsearch is
not fully "unloaded" before the reload.

So I quickly get OOM errors.

When I look at the running threads, I can see at the first launch one
occurrence of

Daemon Thread [org.elasticsearch.common.base.internal.Finalizer] (Running)

After one restart, I can see two of them, and after the third restart, a
third occurrence.

It seems to be the only Thread that remains after a Client.close() and
Node.close().

Any idea ? Do I have to call another ES method to "destroy" all the ES
threads ???

Anything I can do to add more details about this ?

Thanks,

David.

plaflamme · January 5, 2012, 4:03am

Seems to be the Guava "Finalizer" thread issue:

github.com/google/guava

FinalizableReferenceQueue keeps ClassLoaders around.

opened 05:04PM - 31 Oct 14 UTC

closed 02:07AM - 01 Nov 14 UTC

gissuebot

type=defect status=fixed package=base

_[Original issue](https://code.google.com/p/guava-libraries/issues/detail?id=92)… created by **nairb774** on 2008-07-28 at 05:39 PM_ --- If the class loader which loaded the google-collections jar is let go (consider redeploying a war to Apache Geronimo) the class loader will not be able to be garbage collected. This is caused by the thread started by FinalizableReferenceQueue never ending. In other words the garbage collection (for the class loader) can only happen when the thread stops. The solution is to stop the thread when there are no more objects on the queue. This will happen eventually if the rest of the app behaves correctly by not placing state outside of its class loader's environment. Attached is a version of the class which I have used to this effect. This is also a problem in the OSGi space where every jar gets its own class loader and they can be refreshed easily at runtime to do rolling upgrades/bug fixes.

Philippe

On Wed, Jan 4, 2012 at 15:50, David Pilato david@pilato.fr wrote:

Hi there,****

I’m playing with an embedded ES node in a webapp deployed in a tomcat
container.****

ES Node is injected using Spring.****

ES is 0.18.5****

Tomcat is 7.0.19****

Spring is 3.1.0.RELEASE****

And I’m launching everything in debug mode from Eclipse.****

Everything works fine for the “first launch” but each time tomcat restarts
(because something changes in my workspace), it seems that Elasticsearch is
not fully “unloaded” before the reload.****

So I quickly get OOM errors.****

When I look at the running threads, I can see at the first launch one
occurrence of ****

Daemon Thread [org.elasticsearch.common.base.internal.Finalizer] (Running)

After one restart, I can see two of them, and after the third restart, a
third occurrence.****

It seems to be the only Thread that remains after a Client.close() and
Node.close().****

Any idea ? Do I have to call another ES method to “destroy” all the ES
threads ???****

Anything I can do to add more details about this ?****

Thanks,****

David. ****

Michael_Sick · January 5, 2012, 4:56am

Wow what a thread! I see this behavior as well. Anyone have a workaround?

On Wed, Jan 4, 2012 at 11:03 PM, Philippe Laflamme <
philippe.laflamme@obiba.org> wrote:

Seems to be the Guava "Finalizer" thread issue:

FinalizableReferenceQueue keeps ClassLoaders around. · Issue #92 · google/guava · GitHub

Philippe

On Wed, Jan 4, 2012 at 15:50, David Pilato david@pilato.fr wrote:

Hi there,****

I’m playing with an embedded ES node in a webapp deployed in a tomcat
container.****

ES Node is injected using Spring.****

ES is 0.18.5****

Tomcat is 7.0.19****

Spring is 3.1.0.RELEASE****

And I’m launching everything in debug mode from Eclipse.****

Everything works fine for the “first launch” but each time tomcat
restarts (because something changes in my workspace), it seems that
Elasticsearch is not fully “unloaded” before the reload.****

So I quickly get OOM errors.****

When I look at the running threads, I can see at the first launch one
occurrence of ****

Daemon Thread [org.elasticsearch.common.base.internal.Finalizer] (Running)

After one restart, I can see two of them, and after the third restart, a
third occurrence.****

It seems to be the only Thread that remains after a Client.close() and
Node.close().****

Any idea ? Do I have to call another ES method to “destroy” all the ES
threads ???****

Anything I can do to add more details about this ?****

Thanks,****

David. ****

kimchy · January 5, 2012, 1:55pm

Can you try and set index.cache.query.parser.type to none when you start
the node in its settings? Thats the only place that uses weak cache.

On Thu, Jan 5, 2012 at 6:56 AM, Michael Sick <
michael.sick@serenesoftware.com> wrote:

Wow what a thread! I see this behavior as well. Anyone have a workaround?

On Wed, Jan 4, 2012 at 11:03 PM, Philippe Laflamme <
philippe.laflamme@obiba.org> wrote:

Seems to be the Guava "Finalizer" thread issue:

FinalizableReferenceQueue keeps ClassLoaders around. · Issue #92 · google/guava · GitHub

Philippe

On Wed, Jan 4, 2012 at 15:50, David Pilato david@pilato.fr wrote:

Hi there,****

I’m playing with an embedded ES node in a webapp deployed in a tomcat
container.****

ES Node is injected using Spring.****

ES is 0.18.5****

Tomcat is 7.0.19****

Spring is 3.1.0.RELEASE****

And I’m launching everything in debug mode from Eclipse.****

Everything works fine for the “first launch” but each time tomcat
restarts (because something changes in my workspace), it seems that
Elasticsearch is not fully “unloaded” before the reload.****

So I quickly get OOM errors.****

When I look at the running threads, I can see at the first launch one
occurrence of ****

Daemon Thread [org.elasticsearch.common.base.internal.Finalizer]
(Running)****

After one restart, I can see two of them, and after the third restart, a
third occurrence.****

It seems to be the only Thread that remains after a Client.close() and
Node.close().****

Any idea ? Do I have to call another ES method to “destroy” all the ES
threads ???****

Anything I can do to add more details about this ?****

Thanks,****

David. ****

dadoonet · January 5, 2012, 4:53pm

I put index.cache.query.parser.type=none in the same file where I defined the
clustername and it didn't change the behaviour.
I can always see one, two then 3
threads org.elasticsearch.common.base.internal.Finalizer and I finally get OOM
issues.

Finalizer seems to be one of the first threads launched by ES.

My es property file contains only :
cluster.name=es-search discovery.zen.ping.multicast.enabled=false
index.cache.query.parser.type=none

David.

Le 5 janvier 2012 à 14:55, Shay Banon kimchy@gmail.com a écrit :

Can you try and set index.cache.query.parser.type to none when you start the
node in its settings? Thats the only place that uses weak cache.

On Thu, Jan 5, 2012 at 6:56 AM, Michael Sick<michael.sick@serenesoftware.com
[mailto:michael.sick@serenesoftware.com] >wrote:

Wow what a thread! I see this behavior as well. Anyone have a workaround?

On Wed, Jan 4, 2012 at 11:03 PM, Philippe
Laflamme<philippe.laflamme@obiba.org [mailto:philippe.laflamme@obiba.org]

wrote:

Seems to be the Guava "Finalizer" thread issue:

FinalizableReferenceQueue keeps ClassLoaders around. · Issue #92 · google/guava · GitHub

Philippe

On Wed, Jan 4, 2012 at 15:50, David Pilato<david@pilato.fr
[mailto:david@pilato.fr] >wrote:

Hi there,

I’m playing with an embedded ES node in a webapp deployed in a tomcat
container.
ES Node is injected using Spring.

ES is 0.18.5
Tomcat is 7.0.19
Spring is 3.1.0.RELEASE

And I’m launching everything in debug mode from Eclipse.

Everything works fine for the “first launch” but each time tomcat
restarts (because something changes in my workspace), it seems that
Elasticsearch is not fully “unloaded” before the reload.

So I quickly get OOM errors.

When I look at the running threads, I can see at the first launch one
occurrence of
Daemon Thread [org.elasticsearch.common.base.internal.Finalizer]
(Running)

After one restart, I can see two of them, and after the third restart, a
third occurrence.

It seems to be the only Thread that remains after a Client.close() and
Node.close().

Any idea ? Do I have to call another ES method to “destroy” all the ES
threads ???
Anything I can do to add more details about this ?

Thanks,
David.

--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet

kimchy · January 6, 2012, 11:31am

I will double check in 0.18, master not longer has this problem, btw,
because of different usage of Cache guava (and new versions of guava).

On Thu, Jan 5, 2012 at 6:53 PM, david@pilato.fr david@pilato.fr wrote:

**

I put index.cache.query.parser.type=none in the same file where I defined
the clustername and it didn't change the behaviour.

I can always see one, two then 3
threads org.elasticsearch.common.base.internal.Finalizer and I finally get
OOM issues.

Finalizer seems to be one of the first threads launched by ES.

My es property file contains only :

cluster.name=es-search discovery.zen.ping.multicast.enabled=false

index.cache.query.parser.type=none

David.

Le 5 janvier 2012 à 14:55, Shay Banon kimchy@gmail.com a écrit :

Can you try and set index.cache.query.parser.type to none when you start
the node in its settings? Thats the only place that uses weak cache.

On Thu, Jan 5, 2012 at 6:56 AM, Michael Sick <
michael.sick@serenesoftware.com > wrote:

Wow what a thread! I see this behavior as well. Anyone have a workaround?

On Wed, Jan 4, 2012 at 11:03 PM, Philippe Laflamme <
philippe.laflamme@obiba.org > wrote:

Seems to be the Guava "Finalizer" thread issue:

FinalizableReferenceQueue keeps ClassLoaders around. · Issue #92 · google/guava · GitHub

Philippe

On Wed, Jan 4, 2012 at 15:50, David Pilato < david@pilato.fr > wrote:

Hi there,

I’m playing with an embedded ES node in a webapp deployed in a tomcat
container.

ES Node is injected using Spring.

ES is 0.18.5

Tomcat is 7.0.19

Spring is 3.1.0.RELEASE

And I’m launching everything in debug mode from Eclipse.

Everything works fine for the “first launch” but each time tomcat restarts
(because something changes in my workspace), it seems that Elasticsearch is
not fully “unloaded” before the reload.

So I quickly get OOM errors.

When I look at the running threads, I can see at the first launch one
occurrence of

Daemon Thread [org.elasticsearch.common.base.internal.Finalizer] (Running)

After one restart, I can see two of them, and after the third restart, a
third occurrence.

It seems to be the only Thread that remains after a Client.close() and
Node.close().

Any idea ? Do I have to call another ES method to “destroy” all the ES
threads ???

Anything I can do to add more details about this ?

Thanks,

David.

--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet

dadoonet · January 6, 2012, 12:42pm

Ok that's fine for me. I can use 0.19.0-SNAPSHOT.

I will test it and write results here.

Thanks
David
@dadoonet

Le 6 janv. 2012 à 12:31, Shay Banon kimchy@gmail.com a écrit :

I will double check in 0.18, master not longer has this problem, btw, because of different usage of Cache guava (and new versions of guava).

On Thu, Jan 5, 2012 at 6:53 PM, david@pilato.fr david@pilato.fr wrote:
I put index.cache.query.parser.type=none in the same file where I defined the clustername and it didn't change the behaviour.
I can always see one, two then 3 threads org.elasticsearch.common.base.internal.Finalizer and I finally get OOM issues.

Finalizer seems to be one of the first threads launched by ES.

My es property file contains only :
cluster.name=es-search discovery.zen.ping.multicast.enabled=false

index.cache.query.parser.type=none

David.

Le 5 janvier 2012 à 14:55, Shay Banon kimchy@gmail.com a écrit :

Can you try and set index.cache.query.parser.type to none when you start the node in its settings? Thats the only place that uses weak cache.

On Thu, Jan 5, 2012 at 6:56 AM, Michael Sick < michael.sick@serenesoftware.com > wrote:
Wow what a thread! I see this behavior as well. Anyone have a workaround?

On Wed, Jan 4, 2012 at 11:03 PM, Philippe Laflamme < philippe.laflamme@obiba.org > wrote:
Seems to be the Guava "Finalizer" thread issue:

FinalizableReferenceQueue keeps ClassLoaders around. · Issue #92 · google/guava · GitHub

Philippe

On Wed, Jan 4, 2012 at 15:50, David Pilato < david@pilato.fr > wrote:
Hi there,

I’m playing with an embedded ES node in a webapp deployed in a tomcat container.

ES Node is injected using Spring.

ES is 0.18.5

Tomcat is 7.0.19

Spring is 3.1.0.RELEASE

And I’m launching everything in debug mode from Eclipse.

Everything works fine for the “first launch” but each time tomcat restarts (because something changes in my workspace), it seems that Elasticsearch is not fully “unloaded” before the reload.

So I quickly get OOM errors.

When I look at the running threads, I can see at the first launch one occurrence of

Daemon Thread [org.elasticsearch.common.base.internal.Finalizer] (Running)

After one restart, I can see two of them, and after the third restart, a third occurrence.

It seems to be the only Thread that remains after a Client.close() and Node.close().

Any idea ? Do I have to call another ES method to “destroy” all the ES threads ???

Anything I can do to add more details about this ?

Thanks,

David.

--
David Pilato
http://dev.david.pilato.fr/
Twitter : @dadoonet