Potential leak when ES embedded in Tomcat webapp?


(Lukáš Vlček) #1

Hi,

Using ES 0.17.2 in WAR application.

  • Connecting to ES cluster from Tomcat webapp using TransportClient.
  • I restarted the cluster and changed its name which caused the
    TransportClient to be disconnected (that is correct and expected)
  • From this point I started getting the following log message in
    catalina.out every 5 seconds:

[2011-08-02 03:08:55,434] WARN417322[elasticsearch[cached]-pool-1-thread-1]

org.elasticsearch.common.logging.slf4j.Slf4jESLogger.internalWarn(Slf4jESLogger.java:86)

  • [Angel Salvadore] Node [#transport#-1][inet[/x.x.x.x:9300]] not part of
    the cluster Cluster [old_cluster_name], ignoring...

  • I undeployed the WAR application from Tomcat, but...

  • The logging did not stop.

During the undeploy I am getting the following SEVERE message in
catalina.out:

SEVERE: The web application [/search] created a ThreadLocal with key of type
[org.elasticsearch.common.inject.InjectorImpl$1] (value
[org.elasticsearch.common.inject.InjectorImpl$1@6e29991]) and a value of
type [java.lang.Object[]] (value [[Ljava.lang.Object;@6d0021f]) but failed
to remove it when the web application was stopped. Threads are going to be
renewed over time to try and avoid a probable memory leak.
Aug 2, 2011 2:58:08 AM org.apache.catalina.startup.HostConfig checkResources

I make sure that I close the Client during undeploy. (I think Tomcat restart
would help but may be there is something that could be improved in ES
because the above scenario can be easily recreated)

Regards,
Lukas


(Shay Banon) #2

Close will close the thread running on the transport client pinging the
hosts. I just ran a quick test and see that happening... . If you can
recreate it, it would be great.

On Tue, Aug 2, 2011 at 10:19 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Hi,

Using ES 0.17.2 in WAR application.

  • Connecting to ES cluster from Tomcat webapp using TransportClient.
  • I restarted the cluster and changed its name which caused the
    TransportClient to be disconnected (that is correct and expected)
  • From this point I started getting the following log message in
    catalina.out every 5 seconds:

[2011-08-02 03:08:55,434] WARN417322[elasticsearch[cached]-pool-1-thread-1]

org.elasticsearch.common.logging.slf4j.Slf4jESLogger.internalWarn(Slf4jESLogger.java:86)

  • [Angel Salvadore] Node [#transport#-1][inet[/x.x.x.x:9300]] not part of
    the cluster Cluster [old_cluster_name], ignoring...

  • I undeployed the WAR application from Tomcat, but...

  • The logging did not stop.

During the undeploy I am getting the following SEVERE message in
catalina.out:

SEVERE: The web application [/search] created a ThreadLocal with key of
type [org.elasticsearch.common.inject.InjectorImpl$1] (value
[org.elasticsearch.common.inject.InjectorImpl$1@6e29991]) and a value of
type [java.lang.Object[]] (value [[Ljava.lang.Object;@6d0021f]) but failed
to remove it when the web application was stopped. Threads are going to be
renewed over time to try and avoid a probable memory leak.
Aug 2, 2011 2:58:08 AM org.apache.catalina.startup.HostConfig
checkResources

I make sure that I close the Client during undeploy. (I think Tomcat
restart would help but may be there is something that could be improved in
ES because the above scenario can be easily recreated)

Regards,
Lukas


(Lukáš Vlček) #3

You mean preparing a small WAR webapp to allow simple demonstration of this
issue?

On Tue, Aug 2, 2011 at 9:56 AM, Shay Banon kimchy@gmail.com wrote:

Close will close the thread running on the transport client pinging the
hosts. I just ran a quick test and see that happening... . If you can
recreate it, it would be great.

On Tue, Aug 2, 2011 at 10:19 AM, Lukáš Vlček lukas.vlcek@gmail.comwrote:

Hi,

Using ES 0.17.2 in WAR application.

  • Connecting to ES cluster from Tomcat webapp using TransportClient.
  • I restarted the cluster and changed its name which caused the
    TransportClient to be disconnected (that is correct and expected)
  • From this point I started getting the following log message in
    catalina.out every 5 seconds:

[2011-08-02 03:08:55,434]
WARN417322[elasticsearch[cached]-pool-1-thread-1] -
org.elasticsearch.common.logging.slf4j.Slf4jESLogger.internalWarn(Slf4jESLogger.java:86)

  • [Angel Salvadore] Node [#transport#-1][inet[/x.x.x.x:9300]] not part of
    the cluster Cluster [old_cluster_name], ignoring...

  • I undeployed the WAR application from Tomcat, but...

  • The logging did not stop.

During the undeploy I am getting the following SEVERE message in
catalina.out:

SEVERE: The web application [/search] created a ThreadLocal with key of
type [org.elasticsearch.common.inject.InjectorImpl$1] (value
[org.elasticsearch.common.inject.InjectorImpl$1@6e29991]) and a value of
type [java.lang.Object[]] (value [[Ljava.lang.Object;@6d0021f]) but
failed to remove it when the web application was stopped. Threads are going
to be renewed over time to try and avoid a probable memory leak.
Aug 2, 2011 2:58:08 AM org.apache.catalina.startup.HostConfig
checkResources

I make sure that I close the Client during undeploy. (I think Tomcat
restart would help but may be there is something that could be improved in
ES because the above scenario can be easily recreated)

Regards,
Lukas


(Shay Banon) #4

Or even if you can recreate it with a plain Main? If not, and you can
recreate it with a way, its fine as well.

On Tue, Aug 2, 2011 at 11:14 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

You mean preparing a small WAR webapp to allow simple demonstration of this
issue?

On Tue, Aug 2, 2011 at 9:56 AM, Shay Banon kimchy@gmail.com wrote:

Close will close the thread running on the transport client pinging the
hosts. I just ran a quick test and see that happening... . If you can
recreate it, it would be great.

On Tue, Aug 2, 2011 at 10:19 AM, Lukáš Vlček lukas.vlcek@gmail.comwrote:

Hi,

Using ES 0.17.2 in WAR application.

  • Connecting to ES cluster from Tomcat webapp using TransportClient.
  • I restarted the cluster and changed its name which caused the
    TransportClient to be disconnected (that is correct and expected)
  • From this point I started getting the following log message in
    catalina.out every 5 seconds:

[2011-08-02 03:08:55,434]
WARN417322[elasticsearch[cached]-pool-1-thread-1] -
org.elasticsearch.common.logging.slf4j.Slf4jESLogger.internalWarn(Slf4jESLogger.java:86)

  • [Angel Salvadore] Node [#transport#-1][inet[/x.x.x.x:9300]] not part of
    the cluster Cluster [old_cluster_name], ignoring...

  • I undeployed the WAR application from Tomcat, but...

  • The logging did not stop.

During the undeploy I am getting the following SEVERE message in
catalina.out:

SEVERE: The web application [/search] created a ThreadLocal with key of
type [org.elasticsearch.common.inject.InjectorImpl$1] (value
[org.elasticsearch.common.inject.InjectorImpl$1@6e29991]) and a value of
type [java.lang.Object[]] (value [[Ljava.lang.Object;@6d0021f]) but
failed to remove it when the web application was stopped. Threads are going
to be renewed over time to try and avoid a probable memory leak.
Aug 2, 2011 2:58:08 AM org.apache.catalina.startup.HostConfig
checkResources

I make sure that I close the Client during undeploy. (I think Tomcat
restart would help but may be there is something that could be improved in
ES because the above scenario can be easily recreated)

Regards,
Lukas


(Amit Soni) #5

Actually I have been seeing error as well. Unfortunately, it is quite
intermittent and hence tough to reproduce it.

On Tuesday, August 2, 2011 1:30:51 AM UTC-7, kimchy wrote:

Or even if you can recreate it with a plain Main? If not, and you can
recreate it with a way, its fine as well.

On Tue, Aug 2, 2011 at 11:14 AM, Lukáš Vlček <lukas...@gmail.com<javascript:>

wrote:

You mean preparing a small WAR webapp to allow simple demonstration of
this issue?

On Tue, Aug 2, 2011 at 9:56 AM, Shay Banon <kim...@gmail.com<javascript:>

wrote:

Close will close the thread running on the transport client pinging the
hosts. I just ran a quick test and see that happening... . If you can
recreate it, it would be great.

On Tue, Aug 2, 2011 at 10:19 AM, Lukáš Vlček <lukas...@gmail.com<javascript:>

wrote:

Hi,

Using ES 0.17.2 in WAR application.

  • Connecting to ES cluster from Tomcat webapp using TransportClient.
  • I restarted the cluster and changed its name which caused the
    TransportClient to be disconnected (that is correct and expected)
  • From this point I started getting the following log message in
    catalina.out every 5 seconds:

[2011-08-02 03:08:55,434]
WARN417322[elasticsearch[cached]-pool-1-thread-1] -
org.elasticsearch.common.logging.slf4j.Slf4jESLogger.internalWarn(Slf4jESLogger.java:86)

  • [Angel Salvadore] Node [#transport#-1][inet[/x.x.x.x:9300]] not part of
    the cluster Cluster [old_cluster_name], ignoring...

  • I undeployed the WAR application from Tomcat, but...

  • The logging did not stop.

During the undeploy I am getting the following SEVERE message in
catalina.out:

SEVERE: The web application [/search] created a ThreadLocal with key of
type [org.elasticsearch.common.inject.InjectorImpl$1] (value
[org.elasticsearch.common.inject.InjectorImpl$1@6e29991]) and a value of
type [java.lang.Object[]] (value [[Ljava.lang.Object;@6d0021f]) but failed
to remove it when the web application was stopped. Threads are going to be
renewed over time to try and avoid a probable memory leak.
Aug 2, 2011 2:58:08 AM org.apache.catalina.startup.HostConfig
checkResources

I make sure that I close the Client during undeploy. (I think Tomcat
restart would help but may be there is something that could be improved in
ES because the above scenario can be easily recreated)

Regards,
Lukas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4ed58ebc-40b8-42cd-8d64-fc69d86a8e98%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #6

Undeploying the webapp is not enough.
Sadly IIRC due to a very old issue in Guava, all threads are not stopped.
So you have to restart the container.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 28 févr. 2014 à 05:00, "amit.soni" amitsoni29@gmail.com a écrit :

Actually I have been seeing error as well. Unfortunately, it is quite intermittent and hence tough to reproduce it.

On Tuesday, August 2, 2011 1:30:51 AM UTC-7, kimchy wrote:
Or even if you can recreate it with a plain Main? If not, and you can recreate it with a way, its fine as well.

On Tue, Aug 2, 2011 at 11:14 AM, Lukáš Vlček lukas...@gmail.com wrote:
You mean preparing a small WAR webapp to allow simple demonstration of this issue?

On Tue, Aug 2, 2011 at 9:56 AM, Shay Banon kim...@gmail.com wrote:
Close will close the thread running on the transport client pinging the hosts. I just ran a quick test and see that happening... . If you can recreate it, it would be great.

On Tue, Aug 2, 2011 at 10:19 AM, Lukáš Vlček lukas...@gmail.com wrote:
Hi,

Using ES 0.17.2 in WAR application.

  • Connecting to ES cluster from Tomcat webapp using TransportClient.
  • I restarted the cluster and changed its name which caused the TransportClient to be disconnected (that is correct and expected)
  • From this point I started getting the following log message in catalina.out every 5 seconds:

[2011-08-02 03:08:55,434] WARN417322[elasticsearch[cached]-pool-1-thread-1] - org.elasticsearch.common.logging.slf4j.Slf4jESLogger.internalWarn(Slf4jESLogger.java:86) - [Angel Salvadore] Node [#transport#-1][inet[/x.x.x.x:9300]] not part of the cluster Cluster [old_cluster_name], ignoring...

  • I undeployed the WAR application from Tomcat, but...
  • The logging did not stop.

During the undeploy I am getting the following SEVERE message in catalina.out:

SEVERE: The web application [/search] created a ThreadLocal with key of type [org.elasticsearch.common.inject.InjectorImpl$1] (value [org.elasticsearch.common.inject.InjectorImpl$1@6e29991]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@6d0021f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Aug 2, 2011 2:58:08 AM org.apache.catalina.startup.HostConfig checkResources

I make sure that I close the Client during undeploy. (I think Tomcat restart would help but may be there is something that could be improved in ES because the above scenario can be easily recreated)

Regards,
Lukas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4ed58ebc-40b8-42cd-8d64-fc69d86a8e98%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/76D3CD93-3445-4B5D-BEDB-3DD713548543%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.


(Doru Sular) #7

Hi David,

I receive the same message from tomcat with elasticsearch 1.1.0:

SEVERE: The web application [/test-es] created a ThreadLocal with key of

type [org.elasticsearch.common.inject.InjectorImpl$1] (value
[org.elasticsearch.common.inject.InjectorImpl$1@619aaf5]) and a value of
type [java.lang.Object[]] (value [[Ljava.lang.Object;@7f26bdf3]) but failed
to remove it when the web application was stopped. Threads are going to be
renewed over time to try and avoid a probable memory leak.

This cause my web application to fail to redeploy. Restarting tomcat is not
a solution, since in production we will need to have many web applications.
There is no solution for this at all?
Thank you very much,
Doru

On Friday, February 28, 2014 8:19:56 AM UTC+1, David Pilato wrote:

Undeploying the webapp is not enough.
Sadly IIRC due to a very old issue in Guava, all threads are not stopped.
So you have to restart the container.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 28 févr. 2014 à 05:00, "amit.soni" <amits...@gmail.com <javascript:>>
a écrit :

Actually I have been seeing error as well. Unfortunately, it is quite
intermittent and hence tough to reproduce it.

On Tuesday, August 2, 2011 1:30:51 AM UTC-7, kimchy wrote:

Or even if you can recreate it with a plain Main? If not, and you can
recreate it with a way, its fine as well.

On Tue, Aug 2, 2011 at 11:14 AM, Lukáš Vlček lukas...@gmail.com wrote:

You mean preparing a small WAR webapp to allow simple demonstration of
this issue?

On Tue, Aug 2, 2011 at 9:56 AM, Shay Banon kim...@gmail.com wrote:

Close will close the thread running on the transport client pinging the
hosts. I just ran a quick test and see that happening... . If you can
recreate it, it would be great.

On Tue, Aug 2, 2011 at 10:19 AM, Lukáš Vlček lukas...@gmail.comwrote:

Hi,

Using ES 0.17.2 in WAR application.

  • Connecting to ES cluster from Tomcat webapp using TransportClient.
  • I restarted the cluster and changed its name which caused the
    TransportClient to be disconnected (that is correct and expected)
  • From this point I started getting the following log message in
    catalina.out every 5 seconds:

[2011-08-02 03:08:55,434]
WARN417322[elasticsearch[cached]-pool-1-thread-1] -
org.elasticsearch.common.logging.slf4j.Slf4jESLogger.internalWarn(Slf4jESLogger.java:86)

  • [Angel Salvadore] Node [#transport#-1][inet[/x.x.x.x:9300]] not part of
    the cluster Cluster [old_cluster_name], ignoring...

  • I undeployed the WAR application from Tomcat, but...

  • The logging did not stop.

During the undeploy I am getting the following SEVERE message in
catalina.out:

SEVERE: The web application [/search] created a ThreadLocal with key
of type [org.elasticsearch.common.inject.InjectorImpl$1] (value
[org.elasticsearch.common.inject.InjectorImpl$1@6e29991]) and a value of
type [java.lang.Object[]] (value [[Ljava.lang.Object;@6d0021f]) but failed
to remove it when the web application was stopped. Threads are going to be
renewed over time to try and avoid a probable memory leak.
Aug 2, 2011 2:58:08 AM org.apache.catalina.startup.HostConfig
checkResources

I make sure that I close the Client during undeploy. (I think Tomcat
restart would help but may be there is something that could be improved in
ES because the above scenario can be easily recreated)

Regards,
Lukas

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4ed58ebc-40b8-42cd-8d64-fc69d86a8e98%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/789879b5-994b-4439-8e82-5d3c176df558%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #8