StackOverflowError after node connect, java api


(Sebastian Gavarini) #1

Hi all,
I am using ES 0.12.1, java api, local gw, JRockit 1.6, and I am seeing
from time to time in my logs:

Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-1" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-2" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-3" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-4" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-5" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers,
Alexander]clusterService#updateTask-pool-5-thread-1"
java.lang.StackOverflowError

This happens approximately one time every 20 runs of my client code.
It is happening before any of my actual work is started, just after
the cluster join is finished.
The code that joins the cluster in my client side is:

private Node buildNode() {
	return

NodeBuilder.nodeBuilder().client(true).loadConfigSettings(true).node();
}

It's a singleton Guice injected object.
I am using a runtime parameter -Des.network.host=10.0.0.1 for the IP,
and the rest of the configuration is on the file elasticsearch.yml:

cluster:
name: test
transport:
tcp:
port: 5000-5099
connect_timeout: 5s
http:
port: 5200-5299
discovery:
zen:
ping:
multicast:
group: 224.2.2.4
fd:
ping_interval: 5s
ping_timeout: 20s
ping_retries: 3
threadpool:
scaling:
min: 2
max: 10
scheduled_size: 6

gateway:
type: none

Does anybody sees an error that could be causing that behavior?
Thanks,
Sebastian.


(Shay Banon) #2

Hi,

Strange... . Usually a stack overflow exception also includes the stack
trace up to where it happens. I can guess of several reason why this might
happen:

  1. Using jrockit? Can you try and run it with openjdk / sun jdk?
  2. elasticsearch jarjar guice, maybe the fact that Guice is also used messes
    it up?

If you can get to the stack trace of why this happens, it would be much
simpler to analyze it. Maybe you can debug run it, and have an exception
breakpoint on StackOverflowException?

-shay.banon

On Sat, Nov 6, 2010 at 8:32 PM, Sebastian sgavarini@gmail.com wrote:

Hi all,
I am using ES 0.12.1, java api, local gw, JRockit 1.6, and I am seeing
from time to time in my logs:

Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-1" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-2" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-3" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-4" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-5" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers,
Alexander]clusterService#updateTask-pool-5-thread-1"
java.lang.StackOverflowError

This happens approximately one time every 20 runs of my client code.
It is happening before any of my actual work is started, just after
the cluster join is finished.
The code that joins the cluster in my client side is:

   private Node buildNode() {
           return

NodeBuilder.nodeBuilder().client(true).loadConfigSettings(true).node();
}

It's a singleton Guice injected object.
I am using a runtime parameter -Des.network.host=10.0.0.1 for the IP,
and the rest of the configuration is on the file elasticsearch.yml:

cluster:
name: test
transport:
tcp:
port: 5000-5099
connect_timeout: 5s
http:
port: 5200-5299
discovery:
zen:
ping:
multicast:
group: 224.2.2.4
fd:
ping_interval: 5s
ping_timeout: 20s
ping_retries: 3
threadpool:
scaling:
min: 2
max: 10
scheduled_size: 6

gateway:
type: none

Does anybody sees an error that could be causing that behavior?
Thanks,
Sebastian.


(Sebastian Gavarini) #3

Hi Shay,

Thanks for your quick answer. I did a test with debug run as you
suggested, I don't know what you mean by "exception breakpoint", but
when the exception fires the debugger shows the class:

ThreadPoolExecutor$Worker
public void run() {
try {
Runnable task = firstTask;
firstTask = null;
while (task != null || (task = getTask()) != null) {
runTask(task);
task = null;
}
} finally {

           workerDone(this);
        }
    }

line 911

Not very helpful, and all the values in the trace are null or empty,
except backtrace (again I don't know what is that):
StackOverflowError
backtrace = [139779368723408, 139779481643694, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666]

The logs again are:
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-1"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-2"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-3"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-4"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-5"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal]clusterService#updateTask-
pool-5-thread-1" java.lang.StackOverflowError

I am seeing 5 threads again, but in my configuration I have 6 for
scheduled size, and 2-10 for min-max, any idea of where 5 comes from?
what's "tp" pool?

I had to run it like 10 times, but it eventually happened. I am
running Linux Ubuntu Maverick, standard JRockit configuration (no
stack memory messing around), in an AMD X2, (2 cores).

I will try changing JRockit to Sun's, any other idea is more than
welcomed. Nevertheless I am not inclined to think it's the cause, it's
more likely some classloading, constructors, or something like that
race condition.

Regarding Guice, I can't get rid of it that easily, maybe I could just
use your embedded Guice in elasticsearch (removing google's import in
favor of yours), for a quick test if you think it's useful.

Thanks,
Sebastian.

On Nov 6, 4:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Strange... . Usually a stack overflow exception also includes the stack
trace up to where it happens. I can guess of several reason why this might
happen:

  1. Using jrockit? Can you try and run it with openjdk / sun jdk?
  2. elasticsearch jarjar guice, maybe the fact that Guice is also used messes
    it up?

If you can get to the stack trace of why this happens, it would be much
simpler to analyze it. Maybe you can debug run it, and have an exception
breakpoint on StackOverflowException?

-shay.banon

On Sat, Nov 6, 2010 at 8:32 PM, Sebastian sgavar...@gmail.com wrote:

Hi all,
I am using ES 0.12.1, java api, local gw, JRockit 1.6, and I am seeing
from time to time in my logs:

Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-1" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-2" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-3" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-4" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-5" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers,
Alexander]clusterService#updateTask-pool-5-thread-1"
java.lang.StackOverflowError

This happens approximately one time every 20 runs of my client code.
It is happening before any of my actual work is started, just after
the cluster join is finished.
The code that joins the cluster in my client side is:

   private Node buildNode() {
           return

NodeBuilder.nodeBuilder().client(true).loadConfigSettings(true).node();
}

It's a singleton Guice injected object.
I am using a runtime parameter -Des.network.host=10.0.0.1 for the IP,
and the rest of the configuration is on the file elasticsearch.yml:

cluster:
name: test
transport:
tcp:
port: 5000-5099
connect_timeout: 5s
http:
port: 5200-5299
discovery:
zen:
ping:
multicast:
group: 224.2.2.4
fd:
ping_interval: 5s
ping_timeout: 20s
ping_retries: 3
threadpool:
scaling:
min: 2
max: 10
scheduled_size: 6

gateway:
type: none

Does anybody sees an error that could be causing that behavior?
Thanks,
Sebastian.


(Shay Banon) #4

Hi,

An exception breakpoint allows, using the IDE, to break when specific
exception occurs. In IntelliJ, you go over to the Run Menue, click the
Breakpoints option, and the second tab there are exception breakpoints. You
can simply add one for the StackOverflow exception. Then, the importnat bit
here is the debug stack trace, and which methods keep repeating.

If you can try the sun jdk it would be great. Never tried to debug a Stack
Overflow in JRockit, but the ones I saw on the sun jdk do show the stack
trace, and then its easy to understand where it is coming from, as you see
the method(s) that keep repeating.

The [tp] thread pool is the main thread pool in elasticsearch.

-shay.banon

On Sat, Nov 6, 2010 at 11:00 PM, Sebastian sgavarini@gmail.com wrote:

Hi Shay,

Thanks for your quick answer. I did a test with debug run as you
suggested, I don't know what you mean by "exception breakpoint", but
when the exception fires the debugger shows the class:

ThreadPoolExecutor$Worker
public void run() {
try {
Runnable task = firstTask;
firstTask = null;
while (task != null || (task = getTask()) != null) {
runTask(task);
task = null;
}
} finally {

           workerDone(this);
       }
   }

line 911

Not very helpful, and all the values in the trace are null or empty,
except backtrace (again I don't know what is that):
StackOverflowError
backtrace = [139779368723408, 139779481643694, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666]

The logs again are:
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-1"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-2"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-3"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-4"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-5"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal]clusterService#updateTask-
pool-5-thread-1" java.lang.StackOverflowError

I am seeing 5 threads again, but in my configuration I have 6 for
scheduled size, and 2-10 for min-max, any idea of where 5 comes from?
what's "tp" pool?

I had to run it like 10 times, but it eventually happened. I am
running Linux Ubuntu Maverick, standard JRockit configuration (no
stack memory messing around), in an AMD X2, (2 cores).

I will try changing JRockit to Sun's, any other idea is more than
welcomed. Nevertheless I am not inclined to think it's the cause, it's
more likely some classloading, constructors, or something like that
race condition.

Regarding Guice, I can't get rid of it that easily, maybe I could just
use your embedded Guice in elasticsearch (removing google's import in
favor of yours), for a quick test if you think it's useful.

Thanks,
Sebastian.

On Nov 6, 4:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Strange... . Usually a stack overflow exception also includes the stack
trace up to where it happens. I can guess of several reason why this
might
happen:

  1. Using jrockit? Can you try and run it with openjdk / sun jdk?
  2. elasticsearch jarjar guice, maybe the fact that Guice is also used
    messes
    it up?

If you can get to the stack trace of why this happens, it would be much
simpler to analyze it. Maybe you can debug run it, and have an exception
breakpoint on StackOverflowException?

-shay.banon

On Sat, Nov 6, 2010 at 8:32 PM, Sebastian sgavar...@gmail.com wrote:

Hi all,
I am using ES 0.12.1, java api, local gw, JRockit 1.6, and I am seeing
from time to time in my logs:

Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-1" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-2" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-3" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-4" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-5" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers,
Alexander]clusterService#updateTask-pool-5-thread-1"
java.lang.StackOverflowError

This happens approximately one time every 20 runs of my client code.
It is happening before any of my actual work is started, just after
the cluster join is finished.
The code that joins the cluster in my client side is:

   private Node buildNode() {
           return

NodeBuilder.nodeBuilder().client(true).loadConfigSettings(true).node();
}

It's a singleton Guice injected object.
I am using a runtime parameter -Des.network.host=10.0.0.1 for the IP,
and the rest of the configuration is on the file elasticsearch.yml:

cluster:
name: test
transport:
tcp:
port: 5000-5099
connect_timeout: 5s
http:
port: 5200-5299
discovery:
zen:
ping:
multicast:
group: 224.2.2.4
fd:
ping_interval: 5s
ping_timeout: 20s
ping_retries: 3
threadpool:
scaling:
min: 2
max: 10
scheduled_size: 6

gateway:
type: none

Does anybody sees an error that could be causing that behavior?
Thanks,
Sebastian.


(Sebastian Gavarini) #5

Hi Shay,

I tried with open jdk 1.6 and I didn't see the error. Back with
JRockit the exception breakpoint didn't work. Then I did another
simple test, and even though I didn't see the exact same error as
before I think I might have a problem with slf4j.

When trying to run a simple test just joining the cluster from ten
threads:

On Eclipse I see

SLF4J: The following loggers will not work becasue they were created
SLF4J: during the default configuration phase of the underlying
logging system.
SLF4J: See also http://www.slf4j.org/codes.html#substituteLogger
SLF4J: node
SLF4J: node
SLF4J: node
SLF4J: node

On ES server logs I see a couple of:
[2010-11-06 20:23:06,264][INFO ][cluster.service ] [Algrim
the Strong] removed {[Genis-Vell][i0Y0QXoDRbOrDVr2fNFvnw][inet[/
10.0.0.1:5005]]{client=true, data=false},}, reason: zen-disco-
node_failed([Genis-Vell][i0Y0QXoDRbOrDVr2fNFvnw][inet[/10.0.0.1:5005]]
{client=true, data=false}), reason transport disconnected
Exception in thread "elasticsearch[Algrim the
Strong]clusterService#updateTask-pool-5-thread-14"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Algrim the
Strong]clusterService#updateTask-pool-5-thread-15"
java.lang.StackOverflowError

If I remove slf4j from my poms I see on Eclipse:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
further details.
java.lang.NoClassDefFoundError: org/slf4j/impl/StaticLoggerBinder
at org.slf4j.LoggerFactory.staticInitialize(LoggerFactory.java:83)
at org.slf4j.LoggerFactory.(LoggerFactory.java:73)
at
org.elasticsearch.common.logging.slf4j.Slf4jESLoggerFactory.newInstance(Slf4jESLoggerFactory.java:
32)
at
org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:
60)
at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
142)
at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
121)
at
org.elasticsearch.node.internal.InternalNode.(InternalNode.java:
111)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)

I am not sure if this is related but it could, I never used slf4j
before, but read that by adding:

org.slf4j
slf4j-log4j12
true

to my project it should log to log4j.
Am I messing things up? could you please tell me what's the expected
configuration for a java client api?
Thanks,
Sebastian.

On Nov 6, 6:22 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

An exception breakpoint allows, using the IDE, to break when specific
exception occurs. In IntelliJ, you go over to the Run Menue, click the
Breakpoints option, and the second tab there are exception breakpoints. You
can simply add one for the StackOverflow exception. Then, the importnat bit
here is the debug stack trace, and which methods keep repeating.

If you can try the sun jdk it would be great. Never tried to debug a Stack
Overflow in JRockit, but the ones I saw on the sun jdk do show the stack
trace, and then its easy to understand where it is coming from, as you see
the method(s) that keep repeating.

The [tp] thread pool is the main thread pool in elasticsearch.

-shay.banon

On Sat, Nov 6, 2010 at 11:00 PM, Sebastian sgavar...@gmail.com wrote:

Hi Shay,

Thanks for your quick answer. I did a test with debug run as you
suggested, I don't know what you mean by "exception breakpoint", but
when the exception fires the debugger shows the class:

ThreadPoolExecutor$Worker
public void run() {
try {
Runnable task = firstTask;
firstTask = null;
while (task != null || (task = getTask()) != null) {
runTask(task);
task = null;
}
} finally {

           workerDone(this);
       }
   }

line 911

Not very helpful, and all the values in the trace are null or empty,
except backtrace (again I don't know what is that):
StackOverflowError
backtrace = [139779368723408, 139779481643694, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666]

The logs again are:
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-1"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-2"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-3"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-4"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-5"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal]clusterService#updateTask-
pool-5-thread-1" java.lang.StackOverflowError

I am seeing 5 threads again, but in my configuration I have 6 for
scheduled size, and 2-10 for min-max, any idea of where 5 comes from?
what's "tp" pool?

I had to run it like 10 times, but it eventually happened. I am
running Linux Ubuntu Maverick, standard JRockit configuration (no
stack memory messing around), in an AMD X2, (2 cores).

I will try changing JRockit to Sun's, any other idea is more than
welcomed. Nevertheless I am not inclined to think it's the cause, it's
more likely some classloading, constructors, or something like that
race condition.

Regarding Guice, I can't get rid of it that easily, maybe I could just
use your embedded Guice in elasticsearch (removing google's import in
favor of yours), for a quick test if you think it's useful.

Thanks,
Sebastian.

On Nov 6, 4:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Strange... . Usually a stack overflow exception also includes the stack
trace up to where it happens. I can guess of several reason why this
might
happen:

  1. Using jrockit? Can you try and run it with openjdk / sun jdk?
  2. elasticsearch jarjar guice, maybe the fact that Guice is also used
    messes
    it up?

If you can get to the stack trace of why this happens, it would be much
simpler to analyze it. Maybe you can debug run it, and have an exception
breakpoint on StackOverflowException?

-shay.banon

On Sat, Nov 6, 2010 at 8:32 PM, Sebastian sgavar...@gmail.com wrote:

Hi all,
I am using ES 0.12.1, java api, local gw, JRockit 1.6, and I am seeing
from time to time in my logs:

Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-1" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-2" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-3" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-4" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-5" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers,
Alexander]clusterService#updateTask-pool-5-thread-1"
java.lang.StackOverflowError

This happens approximately one time every 20 runs of my client code.
It is happening before any of my actual work is started, just after
the cluster join is finished.
The code that joins the cluster in my client side is:

   private Node buildNode() {
           return

NodeBuilder.nodeBuilder().client(true).loadConfigSettings(true).node();
}

It's a singleton Guice injected object.
I am using a runtime parameter -Des.network.host=10.0.0.1 for the IP,
and the rest of the configuration is on the file elasticsearch.yml:

cluster:
name: test
transport:
tcp:
port: 5000-5099
connect_timeout: 5s
http:
port: 5200-5299
discovery:
zen:
ping:
multicast:
group: 224.2.2.4
fd:
ping_interval: 5s
ping_timeout: 20s
ping_retries: 3
threadpool:
scaling:
min: 2
max: 10
scheduled_size: 6

gateway:
type: none

Does anybody sees an error that could be causing that behavior?
Thanks,
Sebastian.


(Shay Banon) #6

elasticsearch does not have a dependency on slf4j, are you adding it just
because of elasticsearch? Just add log4j to the classpath and see if it
happens (without slf4j, any of its jars), or don't add any logging jars, and
logging will be done using the JDK logging by default.

-shay.banon

On Sun, Nov 7, 2010 at 1:27 AM, Sebastian sgavarini@gmail.com wrote:

Hi Shay,

I tried with open jdk 1.6 and I didn't see the error. Back with
JRockit the exception breakpoint didn't work. Then I did another
simple test, and even though I didn't see the exact same error as
before I think I might have a problem with slf4j.

When trying to run a simple test just joining the cluster from ten
threads:

On Eclipse I see

SLF4J: The following loggers will not work becasue they were created
SLF4J: during the default configuration phase of the underlying
logging system.
SLF4J: See also http://www.slf4j.org/codes.html#substituteLogger
SLF4J: node
SLF4J: node
SLF4J: node
SLF4J: node

On ES server logs I see a couple of:
[2010-11-06 20:23:06,264][INFO ][cluster.service ] [Algrim
the Strong] removed {[Genis-Vell][i0Y0QXoDRbOrDVr2fNFvnw][inet[/
10.0.0.1:5005]]{client=true, data=false},}, reason: zen-disco-
node_failed([Genis-Vell][i0Y0QXoDRbOrDVr2fNFvnw][inet[/10.0.0.1:5005]]
{client=true, data=false}), reason transport disconnected
Exception in thread "elasticsearch[Algrim the
Strong]clusterService#updateTask-pool-5-thread-14"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Algrim the
Strong]clusterService#updateTask-pool-5-thread-15"
java.lang.StackOverflowError

If I remove slf4j from my poms I see on Eclipse:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
further details.
java.lang.NoClassDefFoundError: org/slf4j/impl/StaticLoggerBinder
at org.slf4j.LoggerFactory.staticInitialize(LoggerFactory.java:83)
at org.slf4j.LoggerFactory.(LoggerFactory.java:73)
at

org.elasticsearch.common.logging.slf4j.Slf4jESLoggerFactory.newInstance(Slf4jESLoggerFactory.java:
32)
at

org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.java:
60)
at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
142)
at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
121)
at
org.elasticsearch.node.internal.InternalNode.(InternalNode.java:
111)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)

I am not sure if this is related but it could, I never used slf4j
before, but read that by adding:

org.slf4j
slf4j-log4j12
true

to my project it should log to log4j.
Am I messing things up? could you please tell me what's the expected
configuration for a java client api?
Thanks,
Sebastian.

On Nov 6, 6:22 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

An exception breakpoint allows, using the IDE, to break when specific
exception occurs. In IntelliJ, you go over to the Run Menue, click the
Breakpoints option, and the second tab there are exception breakpoints.
You
can simply add one for the StackOverflow exception. Then, the importnat
bit
here is the debug stack trace, and which methods keep repeating.

If you can try the sun jdk it would be great. Never tried to debug a
Stack
Overflow in JRockit, but the ones I saw on the sun jdk do show the stack
trace, and then its easy to understand where it is coming from, as you
see
the method(s) that keep repeating.

The [tp] thread pool is the main thread pool in elasticsearch.

-shay.banon

On Sat, Nov 6, 2010 at 11:00 PM, Sebastian sgavar...@gmail.com wrote:

Hi Shay,

Thanks for your quick answer. I did a test with debug run as you
suggested, I don't know what you mean by "exception breakpoint", but
when the exception fires the debugger shows the class:

ThreadPoolExecutor$Worker
public void run() {
try {
Runnable task = firstTask;
firstTask = null;
while (task != null || (task = getTask()) != null) {
runTask(task);
task = null;
}
} finally {

           workerDone(this);
       }
   }

line 911

Not very helpful, and all the values in the trace are null or empty,
except backtrace (again I don't know what is that):
StackOverflowError
backtrace = [139779368723408, 139779481643694, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666]

The logs again are:
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-1"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-2"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-3"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-4"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-5"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal]clusterService#updateTask-
pool-5-thread-1" java.lang.StackOverflowError

I am seeing 5 threads again, but in my configuration I have 6 for
scheduled size, and 2-10 for min-max, any idea of where 5 comes from?
what's "tp" pool?

I had to run it like 10 times, but it eventually happened. I am
running Linux Ubuntu Maverick, standard JRockit configuration (no
stack memory messing around), in an AMD X2, (2 cores).

I will try changing JRockit to Sun's, any other idea is more than
welcomed. Nevertheless I am not inclined to think it's the cause, it's
more likely some classloading, constructors, or something like that
race condition.

Regarding Guice, I can't get rid of it that easily, maybe I could just
use your embedded Guice in elasticsearch (removing google's import in
favor of yours), for a quick test if you think it's useful.

Thanks,
Sebastian.

On Nov 6, 4:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Strange... . Usually a stack overflow exception also includes the
stack

trace up to where it happens. I can guess of several reason why this
might
happen:

  1. Using jrockit? Can you try and run it with openjdk / sun jdk?
  2. elasticsearch jarjar guice, maybe the fact that Guice is also used
    messes
    it up?

If you can get to the stack trace of why this happens, it would be
much

simpler to analyze it. Maybe you can debug run it, and have an
exception

breakpoint on StackOverflowException?

-shay.banon

On Sat, Nov 6, 2010 at 8:32 PM, Sebastian sgavar...@gmail.com
wrote:

Hi all,
I am using ES 0.12.1, java api, local gw, JRockit 1.6, and I am
seeing

from time to time in my logs:

Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-1" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-2" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-3" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-4" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-5" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers,
Alexander]clusterService#updateTask-pool-5-thread-1"
java.lang.StackOverflowError

This happens approximately one time every 20 runs of my client
code.

It is happening before any of my actual work is started, just after
the cluster join is finished.
The code that joins the cluster in my client side is:

   private Node buildNode() {
           return

NodeBuilder.nodeBuilder().client(true).loadConfigSettings(true).node();

   }

It's a singleton Guice injected object.
I am using a runtime parameter -Des.network.host=10.0.0.1 for the
IP,

and the rest of the configuration is on the file elasticsearch.yml:

cluster:
name: test
transport:
tcp:
port: 5000-5099
connect_timeout: 5s
http:
port: 5200-5299
discovery:
zen:
ping:
multicast:
group: 224.2.2.4
fd:
ping_interval: 5s
ping_timeout: 20s
ping_retries: 3
threadpool:
scaling:
min: 2
max: 10
scheduled_size: 6

gateway:
type: none

Does anybody sees an error that could be causing that behavior?
Thanks,
Sebastian.


(Sebastian Gavarini) #7

You are right, I thought I needed it, I added a couple of new jars to
my project at the same time and one of them was using slf4j, but not
ES, I removed it and the slf4j messages went away.
Unfortunately not the stackOverflow. I connected the remote debugger
to my ES server, and I see a huge stack of:

SendRequestTransportException(RemoteTransportException.fillStack()
line 58
SendRequestTransportException.fillInStackTrace() line 34
...

Does that tells you something? because for me the chain goes up to
Throwable: public synchronized native Throwable fillInStackTrace();

Sebastian.

On Nov 6, 8:59 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

elasticsearch does not have a dependency on slf4j, are you adding it just
because of elasticsearch? Just add log4j to the classpath and see if it
happens (without slf4j, any of its jars), or don't add any logging jars, and
logging will be done using the JDK logging by default.

-shay.banon

On Sun, Nov 7, 2010 at 1:27 AM, Sebastian sgavar...@gmail.com wrote:

Hi Shay,

I tried with open jdk 1.6 and I didn't see the error. Back with
JRockit the exception breakpoint didn't work. Then I did another
simple test, and even though I didn't see the exact same error as
before I think I might have a problem with slf4j.

When trying to run a simple test just joining the cluster from ten
threads:

On Eclipse I see

SLF4J: The following loggers will not work becasue they were created
SLF4J: during the default configuration phase of the underlying
logging system.
SLF4J: See alsohttp://www.slf4j.org/codes.html#substituteLogger
SLF4J: node
SLF4J: node
SLF4J: node
SLF4J: node

On ES server logs I see a couple of:
[2010-11-06 20:23:06,264][INFO ][cluster.service ] [Algrim
the Strong] removed {[Genis-Vell][i0Y0QXoDRbOrDVr2fNFvnw][inet[/
10.0.0.1:5005]]{client=true, data=false},}, reason: zen-disco-
node_failed([Genis-Vell][i0Y0QXoDRbOrDVr2fNFvnw][inet[/10.0.0.1:5005]]
{client=true, data=false}), reason transport disconnected
Exception in thread "elasticsearch[Algrim the
Strong]clusterService#updateTask-pool-5-thread-14"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Algrim the
Strong]clusterService#updateTask-pool-5-thread-15"
java.lang.StackOverflowError

If I remove slf4j from my poms I see on Eclipse:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Seehttp://www.slf4j.org/codes.html#StaticLoggerBinderfor
further details.
java.lang.NoClassDefFoundError: org/slf4j/impl/StaticLoggerBinder
at org.slf4j.LoggerFactory.staticInitialize(LoggerFactory.java:83)
at org.slf4j.LoggerFactory.(LoggerFactory.java:73)
at

org.elasticsearch.common.logging.slf4j.Slf4jESLoggerFactory.newInstance(Slf 4jESLoggerFactory.java:
32)
at

org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory. java:
60)
at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
142)
at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
121)
at
org.elasticsearch.node.internal.InternalNode.(InternalNode.java:
111)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)

I am not sure if this is related but it could, I never used slf4j
before, but read that by adding:

org.slf4j
slf4j-log4j12
true

to my project it should log to log4j.
Am I messing things up? could you please tell me what's the expected
configuration for a java client api?
Thanks,
Sebastian.

On Nov 6, 6:22 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

An exception breakpoint allows, using the IDE, to break when specific
exception occurs. In IntelliJ, you go over to the Run Menue, click the
Breakpoints option, and the second tab there are exception breakpoints.
You
can simply add one for the StackOverflow exception. Then, the importnat
bit
here is the debug stack trace, and which methods keep repeating.

If you can try the sun jdk it would be great. Never tried to debug a
Stack
Overflow in JRockit, but the ones I saw on the sun jdk do show the stack
trace, and then its easy to understand where it is coming from, as you
see
the method(s) that keep repeating.

The [tp] thread pool is the main thread pool in elasticsearch.

-shay.banon

On Sat, Nov 6, 2010 at 11:00 PM, Sebastian sgavar...@gmail.com wrote:

Hi Shay,

Thanks for your quick answer. I did a test with debug run as you
suggested, I don't know what you mean by "exception breakpoint", but
when the exception fires the debugger shows the class:

ThreadPoolExecutor$Worker
public void run() {
try {
Runnable task = firstTask;
firstTask = null;
while (task != null || (task = getTask()) != null) {
runTask(task);
task = null;
}
} finally {

           workerDone(this);
       }
   }

line 911

Not very helpful, and all the values in the trace are null or empty,
except backtrace (again I don't know what is that):
StackOverflowError
backtrace = [139779368723408, 139779481643694, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666]

The logs again are:
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-1"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-2"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-3"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-4"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-5"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal]clusterService#updateTask-
pool-5-thread-1" java.lang.StackOverflowError

I am seeing 5 threads again, but in my configuration I have 6 for
scheduled size, and 2-10 for min-max, any idea of where 5 comes from?
what's "tp" pool?

I had to run it like 10 times, but it eventually happened. I am
running Linux Ubuntu Maverick, standard JRockit configuration (no
stack memory messing around), in an AMD X2, (2 cores).

I will try changing JRockit to Sun's, any other idea is more than
welcomed. Nevertheless I am not inclined to think it's the cause, it's
more likely some classloading, constructors, or something like that
race condition.

Regarding Guice, I can't get rid of it that easily, maybe I could just
use your embedded Guice in elasticsearch (removing google's import in
favor of yours), for a quick test if you think it's useful.

Thanks,
Sebastian.

On Nov 6, 4:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Strange... . Usually a stack overflow exception also includes the
stack

trace up to where it happens. I can guess of several reason why this
might
happen:

  1. Using jrockit? Can you try and run it with openjdk / sun jdk?
  2. elasticsearch jarjar guice, maybe the fact that Guice is also used
    messes
    it up?

If you can get to the stack trace of why this happens, it would be
much

simpler to analyze it. Maybe you can debug run it, and have an
exception

breakpoint on StackOverflowException?

-shay.banon

On Sat, Nov 6, 2010 at 8:32 PM, Sebastian sgavar...@gmail.com
wrote:

Hi all,
I am using ES 0.12.1, java api, local gw, JRockit 1.6, and I am
seeing

from time to time in my logs:

Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-1" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-2" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-3" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-4" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers, Alexander][tp]-pool-1-
thread-5" java.lang.StackOverflowError
Exception in thread "elasticsearch[Summers,
Alexander]clusterService#updateTask-pool-5-thread-1"
java.lang.StackOverflowError

This happens approximately one time every 20 runs of my client
code.

It is happening before any of my actual work is started, just after
the cluster join is finished.
The code that joins the cluster in my client side is:

   private Node buildNode() {
           return

NodeBuilder.nodeBuilder().client(true).loadConfigSettings(true).node();

   }

It's a

...

read more »


(Sebastian Gavarini) #8

I was able to get some more info with the remote debugger, my client
code runs two threads with:
Node node =
NodeBuilder.nodeBuilder().client(true).loadConfigSettings(true).node();
Client client = node.client();
System.out.println("got up to here");
client.close();

And I am seeing the "got up to here" messages.
The remote ES server stacks are:

Daemon Thread [elasticsearch[Sofen, Karla]clusterService#updateTask-
pool-5-thread-1] (Suspended (breakpoint at line 39 in
RemoteTransportException))
NodeNotConnectedException(RemoteTransportException).(String,
TransportAddress, String, Throwable) line: 39

NodeNotConnectedException(ConnectTransportException).(DiscoveryNode,
String, String, Throwable) line: 44

NodeNotConnectedException(ConnectTransportException).(DiscoveryNode,
String) line: 32
NodeNotConnectedException.(DiscoveryNode, String) line: 32
NettyTransport.nodeChannel(DiscoveryNode) line: 473
NettyTransport.sendRequest(DiscoveryNode, long, String, Streamable,
TransportRequestOptions) line: 391
TransportService.sendRequest(DiscoveryNode, String, Streamable,
TransportRequestOptions, TransportResponseHandler) line: 182
TransportService.sendRequest(DiscoveryNode, String, Streamable,
TransportResponseHandler) line: 170
PublishClusterStateAction.publish(ClusterState) line: 70
ZenDiscovery.publish(ClusterState) line: 221
DiscoveryService.publish(ClusterState) line: 110
InternalClusterService$2.run() line: 221
ThreadPoolExecutor$Worker.runTask(Runnable) line: 885
ThreadPoolExecutor$Worker.run() line: 907
Thread.run() line: 619

Daemon Thread [elasticsearch[Sofen, Karla]clusterService#updateTask-
pool-5-thread-1] (Suspended (breakpoint at line 58 in
RemoteTransportException))
SendRequestTransportException(RemoteTransportException).fillStack()
line: 58
SendRequestTransportException.fillInStackTrace() line: 34
SendRequestTransportException(RemoteTransportException).fillStack()
line: 58
SendRequestTransportException.fillInStackTrace() line: 34
SendRequestTransportException(Throwable).(String, Throwable)
line: 218
SendRequestTransportException(Exception).(String, Throwable)
line: 59
SendRequestTransportException(RuntimeException).(String,
Throwable) line: 61
SendRequestTransportException(ElasticSearchException).(String,
Throwable) line: 46
SendRequestTransportException(TransportException).(String,
Throwable) line: 34

SendRequestTransportException(RemoteTransportException).(String,
TransportAddress, String, Throwable) line: 39
SendRequestTransportException.(DiscoveryNode, String,
Throwable) line: 30
TransportService.sendRequest(DiscoveryNode, String, Streamable,
TransportRequestOptions, TransportResponseHandler) line: 197
TransportService.sendRequest(DiscoveryNode, String, Streamable,
TransportResponseHandler) line: 170
PublishClusterStateAction.publish(ClusterState) line: 70
ZenDiscovery.publish(ClusterState) line: 221
DiscoveryService.publish(ClusterState) line: 110
InternalClusterService$2.run() line: 221
ThreadPoolExecutor$Worker.runTask(Runnable) line: 885
ThreadPoolExecutor$Worker.run() line: 907
Thread.run() line: 619

I got the first one of them, hit continue a couple of times, and then
got the second one.
You can see the offending lines:
SendRequestTransportException(RemoteTransportException).fillStack()
line: 58
SendRequestTransportException.fillInStackTrace() line: 34

Do you want me to get some value from any variable?

On Nov 6, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

You are right, I thought I needed it, I added a couple of new jars to
my project at the same time and one of them was using slf4j, but not
ES, I removed it and the slf4j messages went away.
Unfortunately not the stackOverflow. I connected the remote debugger
to my ES server, and I see a huge stack of:

SendRequestTransportException(RemoteTransportException.fillStack()
line 58
SendRequestTransportException.fillInStackTrace() line 34
...

Does that tells you something? because for me the chain goes up to
Throwable: public synchronized native Throwable fillInStackTrace();

Sebastian.

On Nov 6, 8:59 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

elasticsearch does not have a dependency on slf4j, are you adding it just
because of elasticsearch? Just add log4j to the classpath and see if it
happens (without slf4j, any of its jars), or don't add any logging jars, and
logging will be done using the JDK logging by default.

-shay.banon

On Sun, Nov 7, 2010 at 1:27 AM, Sebastian sgavar...@gmail.com wrote:

Hi Shay,

I tried with open jdk 1.6 and I didn't see the error. Back with
JRockit the exception breakpoint didn't work. Then I did another
simple test, and even though I didn't see the exact same error as
before I think I might have a problem with slf4j.

When trying to run a simple test just joining the cluster from ten
threads:

On Eclipse I see

SLF4J: The following loggers will not work becasue they were created
SLF4J: during the default configuration phase of the underlying
logging system.
SLF4J: See alsohttp://www.slf4j.org/codes.html#substituteLogger
SLF4J: node
SLF4J: node
SLF4J: node
SLF4J: node

On ES server logs I see a couple of:
[2010-11-06 20:23:06,264][INFO ][cluster.service ] [Algrim
the Strong] removed {[Genis-Vell][i0Y0QXoDRbOrDVr2fNFvnw][inet[/
10.0.0.1:5005]]{client=true, data=false},}, reason: zen-disco-
node_failed([Genis-Vell][i0Y0QXoDRbOrDVr2fNFvnw][inet[/10.0.0.1:5005]]
{client=true, data=false}), reason transport disconnected
Exception in thread "elasticsearch[Algrim the
Strong]clusterService#updateTask-pool-5-thread-14"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Algrim the
Strong]clusterService#updateTask-pool-5-thread-15"
java.lang.StackOverflowError

If I remove slf4j from my poms I see on Eclipse:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Seehttp://www.slf4j.org/codes.html#StaticLoggerBinderfor
further details.
java.lang.NoClassDefFoundError: org/slf4j/impl/StaticLoggerBinder
at org.slf4j.LoggerFactory.staticInitialize(LoggerFactory.java:83)
at org.slf4j.LoggerFactory.(LoggerFactory.java:73)
at

org.elasticsearch.common.logging.slf4j.Slf4jESLoggerFactory.newInstance(Slf 4jESLoggerFactory.java:
32)
at

org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory. java:
60)
at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
142)
at org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
121)
at
org.elasticsearch.node.internal.InternalNode.(InternalNode.java:
111)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)

I am not sure if this is related but it could, I never used slf4j
before, but read that by adding:

org.slf4j
slf4j-log4j12
true

to my project it should log to log4j.
Am I messing things up? could you please tell me what's the expected
configuration for a java client api?
Thanks,
Sebastian.

On Nov 6, 6:22 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

An exception breakpoint allows, using the IDE, to break when specific
exception occurs. In IntelliJ, you go over to the Run Menue, click the
Breakpoints option, and the second tab there are exception breakpoints.
You
can simply add one for the StackOverflow exception. Then, the importnat
bit
here is the debug stack trace, and which methods keep repeating.

If you can try the sun jdk it would be great. Never tried to debug a
Stack
Overflow in JRockit, but the ones I saw on the sun jdk do show the stack
trace, and then its easy to understand where it is coming from, as you
see
the method(s) that keep repeating.

The [tp] thread pool is the main thread pool in elasticsearch.

-shay.banon

On Sat, Nov 6, 2010 at 11:00 PM, Sebastian sgavar...@gmail.com wrote:

Hi Shay,

Thanks for your quick answer. I did a test with debug run as you
suggested, I don't know what you mean by "exception breakpoint", but
when the exception fires the debugger shows the class:

ThreadPoolExecutor$Worker
public void run() {
try {
Runnable task = firstTask;
firstTask = null;
while (task != null || (task = getTask()) != null) {
runTask(task);
task = null;
}
} finally {

           workerDone(this);
       }
   }

line 911

Not very helpful, and all the values in the trace are null or empty,
except backtrace (again I don't know what is that):
StackOverflowError
backtrace = [139779368723408, 139779481643694, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666, 139779471734800, 139779478653446, 139779368723408,
139779481643698, 139779446576800, 139779481643666, 139779471734800,
139779478653446, 139779368723408, 139779481643698, 139779446576800,
139779481643666]

The logs again are:
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-1"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-2"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-3"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-4"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-5"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal]clusterService#updateTask-
pool-5-thread-1" java.lang.StackOverflowError

I am seeing 5 threads again, but in my configuration I have 6 for
scheduled size, and 2-10 for min-max, any idea of where 5 comes from?
what's "tp" pool?

I had to run it like 10 times, but it eventually happened. I am
running Linux Ubuntu Maverick, standard JRockit configuration (no
stack memory messing around), in an AMD X2, (2 cores).

I will try changing JRockit to Sun's, any other idea is more than
welcomed. Nevertheless I am not inclined to think it's the cause, it's
more likely some classloading, constructors, or something like that
race condition.

Regarding Guice, I can't get rid of it that easily, maybe I could just
use your embedded Guice in elasticsearch (removing google's import in
favor of yours), for a quick test if you think it's useful.

Thanks,
Sebastian.

On Nov 6, 4:19 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Strange... . Usually a stack overflow exception also includes the
stack

trace up to where it happens. I can guess of several reason why this
might
happen:

  1. Using jrockit? Can you try and run it with openjdk / sun jdk?
  2. elasticsearch jarjar guice, maybe the fact that Guice is also used
    messes
    it up?

If you can get to the stack trace of why this happens, it would be
much

simpler to analyze it. Maybe you can debug run it, and have an
exception

breakpoint on StackOverflowException?

-shay.banon

On Sat, Nov 6, 2010 at 8:32 PM, Sebastian sgavar...@gmail.com
wrote:

Hi all,
I am using...

read more »


(Shay Banon) #9

Hi,

Strange, seems like the (based on what you posted) that the stack
overflow comes from how filling stack trace is controlled by the transport
exceptions in elasticsearch. I have pushed better handling in this case,
lets see if it solves it. Its in master, so you will need to get
master/latest, build a distribution (simply run gradlew release, the
distribution will be under build/distributions). Remember to also add the
updated elasticsearch jar file to your client.

-shay.banon

On Sun, Nov 7, 2010 at 2:54 AM, Sebastian sgavarini@gmail.com wrote:

I was able to get some more info with the remote debugger, my client
code runs two threads with:
Node node =
NodeBuilder.nodeBuilder().client(true).loadConfigSettings(true).node();
Client client = node.client();
System.out.println("got up to here");
client.close();

And I am seeing the "got up to here" messages.
The remote ES server stacks are:

Daemon Thread [elasticsearch[Sofen, Karla]clusterService#updateTask-
pool-5-thread-1] (Suspended (breakpoint at line 39 in
RemoteTransportException))
NodeNotConnectedException(RemoteTransportException).(String,
TransportAddress, String, Throwable) line: 39

NodeNotConnectedException(ConnectTransportException).(DiscoveryNode,
String, String, Throwable) line: 44

NodeNotConnectedException(ConnectTransportException).(DiscoveryNode,
String) line: 32
NodeNotConnectedException.(DiscoveryNode, String) line: 32
NettyTransport.nodeChannel(DiscoveryNode) line: 473
NettyTransport.sendRequest(DiscoveryNode, long, String, Streamable,
TransportRequestOptions) line: 391
TransportService.sendRequest(DiscoveryNode, String, Streamable,
TransportRequestOptions, TransportResponseHandler) line: 182
TransportService.sendRequest(DiscoveryNode, String, Streamable,
TransportResponseHandler) line: 170
PublishClusterStateAction.publish(ClusterState) line: 70
ZenDiscovery.publish(ClusterState) line: 221
DiscoveryService.publish(ClusterState) line: 110
InternalClusterService$2.run() line: 221
ThreadPoolExecutor$Worker.runTask(Runnable) line: 885
ThreadPoolExecutor$Worker.run() line: 907
Thread.run() line: 619

Daemon Thread [elasticsearch[Sofen, Karla]clusterService#updateTask-
pool-5-thread-1] (Suspended (breakpoint at line 58 in
RemoteTransportException))
SendRequestTransportException(RemoteTransportException).fillStack()
line: 58
SendRequestTransportException.fillInStackTrace() line: 34
SendRequestTransportException(RemoteTransportException).fillStack()
line: 58
SendRequestTransportException.fillInStackTrace() line: 34
SendRequestTransportException(Throwable).(String, Throwable)
line: 218
SendRequestTransportException(Exception).(String, Throwable)
line: 59
SendRequestTransportException(RuntimeException).(String,
Throwable) line: 61
SendRequestTransportException(ElasticSearchException).(String,
Throwable) line: 46
SendRequestTransportException(TransportException).(String,
Throwable) line: 34

SendRequestTransportException(RemoteTransportException).(String,
TransportAddress, String, Throwable) line: 39
SendRequestTransportException.(DiscoveryNode, String,
Throwable) line: 30
TransportService.sendRequest(DiscoveryNode, String, Streamable,
TransportRequestOptions, TransportResponseHandler) line: 197
TransportService.sendRequest(DiscoveryNode, String, Streamable,
TransportResponseHandler) line: 170
PublishClusterStateAction.publish(ClusterState) line: 70
ZenDiscovery.publish(ClusterState) line: 221
DiscoveryService.publish(ClusterState) line: 110
InternalClusterService$2.run() line: 221
ThreadPoolExecutor$Worker.runTask(Runnable) line: 885
ThreadPoolExecutor$Worker.run() line: 907
Thread.run() line: 619

I got the first one of them, hit continue a couple of times, and then
got the second one.
You can see the offending lines:
SendRequestTransportException(RemoteTransportException).fillStack()
line: 58
SendRequestTransportException.fillInStackTrace() line: 34

Do you want me to get some value from any variable?

On Nov 6, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

You are right, I thought I needed it, I added a couple of new jars to
my project at the same time and one of them was using slf4j, but not
ES, I removed it and the slf4j messages went away.
Unfortunately not the stackOverflow. I connected the remote debugger
to my ES server, and I see a huge stack of:

SendRequestTransportException(RemoteTransportException.fillStack()
line 58
SendRequestTransportException.fillInStackTrace() line 34
...

Does that tells you something? because for me the chain goes up to
Throwable: public synchronized native Throwable fillInStackTrace();

Sebastian.

On Nov 6, 8:59 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

elasticsearch does not have a dependency on slf4j, are you adding it
just

because of elasticsearch? Just add log4j to the classpath and see if it
happens (without slf4j, any of its jars), or don't add any logging
jars, and

logging will be done using the JDK logging by default.

-shay.banon

On Sun, Nov 7, 2010 at 1:27 AM, Sebastian sgavar...@gmail.com wrote:

Hi Shay,

I tried with open jdk 1.6 and I didn't see the error. Back with
JRockit the exception breakpoint didn't work. Then I did another
simple test, and even though I didn't see the exact same error as
before I think I might have a problem with slf4j.

When trying to run a simple test just joining the cluster from ten
threads:

On Eclipse I see

SLF4J: The following loggers will not work becasue they were created
SLF4J: during the default configuration phase of the underlying
logging system.
SLF4J: See alsohttp://www.slf4j.org/codes.html#substituteLogger
SLF4J: node
SLF4J: node
SLF4J: node
SLF4J: node

On ES server logs I see a couple of:
[2010-11-06 20:23:06,264][INFO ][cluster.service ] [Algrim
the Strong] removed {[Genis-Vell][i0Y0QXoDRbOrDVr2fNFvnw][inet[/
10.0.0.1:5005]]{client=true, data=false},}, reason: zen-disco-
node_failed([Genis-Vell][i0Y0QXoDRbOrDVr2fNFvnw][inet[/10.0.0.1:5005
]]

{client=true, data=false}), reason transport disconnected
Exception in thread "elasticsearch[Algrim the
Strong]clusterService#updateTask-pool-5-thread-14"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Algrim the
Strong]clusterService#updateTask-pool-5-thread-15"
java.lang.StackOverflowError

If I remove slf4j from my poms I see on Eclipse:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Seehttp://www.slf4j.org/codes.html#StaticLoggerBinderfor
further details.
java.lang.NoClassDefFoundError: org/slf4j/impl/StaticLoggerBinder
at
org.slf4j.LoggerFactory.staticInitialize(LoggerFactory.java:83)

   at org.slf4j.LoggerFactory.<clinit>(LoggerFactory.java:73)
   at

org.elasticsearch.common.logging.slf4j.Slf4jESLoggerFactory.newInstance(Slf
4jESLoggerFactory.java:

  1. at

org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.
java:

  1. at
    org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
  1. at
    org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
  1. at

org.elasticsearch.node.internal.InternalNode.(InternalNode.java:

  1. at
    org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
   at

org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)

I am not sure if this is related but it could, I never used slf4j
before, but read that by adding:

org.slf4j
slf4j-log4j12
true

to my project it should log to log4j.
Am I messing things up? could you please tell me what's the expected
configuration for a java client api?
Thanks,
Sebastian.

On Nov 6, 6:22 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

An exception breakpoint allows, using the IDE, to break when
specific

exception occurs. In IntelliJ, you go over to the Run Menue, click
the

Breakpoints option, and the second tab there are exception
breakpoints.

You

can simply add one for the StackOverflow exception. Then, the
importnat

bit

here is the debug stack trace, and which methods keep repeating.

If you can try the sun jdk it would be great. Never tried to
debug a

Stack

Overflow in JRockit, but the ones I saw on the sun jdk do show the
stack

trace, and then its easy to understand where it is coming from, as
you

see

the method(s) that keep repeating.

The [tp] thread pool is the main thread pool in elasticsearch.

-shay.banon

On Sat, Nov 6, 2010 at 11:00 PM, Sebastian sgavar...@gmail.com
wrote:

Hi Shay,

Thanks for your quick answer. I did a test with debug run as you
suggested, I don't know what you mean by "exception breakpoint",
but

when the exception fires the debugger shows the class:

ThreadPoolExecutor$Worker
public void run() {
try {
Runnable task = firstTask;
firstTask = null;
while (task != null || (task = getTask()) != null)
{

               runTask(task);
               task = null;
           }
       } finally {
           workerDone(this);
       }
   }

line 911

Not very helpful, and all the values in the trace are null or
empty,

except backtrace (again I don't know what is that):
StackOverflowError
backtrace = [139779368723408, 139779481643694, 139779446576800,
139779481643666, 139779471734800, 139779478653446,
139779368723408,

139779481643698, 139779446576800, 139779481643666,
139779471734800,

139779478653446, 139779368723408, 139779481643698,
139779446576800,

139779481643666, 139779471734800, 139779478653446,
139779368723408,

139779481643698, 139779446576800, 139779481643666,
139779471734800,

139779478653446, 139779368723408, 139779481643698,
139779446576800,

139779481643666, 139779471734800, 139779478653446,
139779368723408,

139779481643698, 139779446576800, 139779481643666,
139779471734800,

139779478653446, 139779368723408, 139779481643698,
139779446576800,

139779481643666, 139779471734800, 139779478653446,
139779368723408,

139779481643698, 139779446576800, 139779481643666,
139779471734800,

139779478653446, 139779368723408, 139779481643698,
139779446576800,

139779481643666, 139779471734800, 139779478653446,
139779368723408,

139779481643698, 139779446576800, 139779481643666,
139779471734800,

139779478653446, 139779368723408, 139779481643698,
139779446576800,

139779481643666]

The logs again are:
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-1"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-2"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-3"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-4"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Portal][tp]-pool-1-thread-5"
java.lang.StackOverflowError
Exception in thread
"elasticsearch[Portal]clusterService#updateTask-

pool-5-thread-1" java.lang.StackOverflowError

I am seeing 5 threads again, but in my configuration I have 6 for
scheduled size, and 2-10 for min-max, any idea of where 5 comes
from?

what's "tp" pool?

I had to run it like 10 times, but it eventually happened. I am
running Linux Ubuntu Maverick, standard JRockit configuration (no
stack memory messing around), in an AMD X2, (2 cores).

I will try changing JRockit to Sun's, any other idea is more than
welcomed. Nevertheless I am not inclined to think it's the cause,
it's

more likely some classloading, constructors, or something like
that

race condition.

Regarding Guice, I can't get rid of it that easily, maybe I could
just

use your embedded Guice in elasticsearch (removing google's
import in

favor of yours), for a quick test if you think it's useful.

Thanks,
Sebastian.

On Nov 6, 4:19 pm, Shay Banon shay.ba...@elasticsearch.com
wrote:

Hi,

Strange... . Usually a stack overflow exception also includes
the

stack

trace up to where it happens. I can guess of several reason why
this

might

happen:

  1. Using jrockit? Can you try and run it with openjdk / sun
    jdk?
  1. elasticsearch jarjar guice, maybe the fact that Guice is
    also used

messes

it up?

If you can get to the stack trace of why this happens, it
would be

much

simpler to analyze it. Maybe you can debug run it, and have an
exception

breakpoint on StackOverflowException?

-shay.banon

On Sat, Nov 6, 2010 at 8:32 PM, Sebastian <sgavar...@gmail.com

wrote:

Hi all,
I am using...

read more »


(Sebastian Gavarini) #10

Hi Shay,

Yep, you nailed it!, I run my test many times against master, with 2
up to 20 threads in JRockit and now it's working fine.

Thanks a lot,
Sebastian.

On Nov 6, 11:10 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Strange, seems like the (based on what you posted) that the stack
overflow comes from how filling stack trace is controlled by the transport
exceptions in elasticsearch. I have pushed better handling in this case,
lets see if it solves it. Its in master, so you will need to get
master/latest, build a distribution (simply run gradlew release, the
distribution will be under build/distributions). Remember to also add the
updated elasticsearch jar file to your client.

-shay.banon

On Sun, Nov 7, 2010 at 2:54 AM, Sebastian sgavar...@gmail.com wrote:

I was able to get some more info with the remote debugger, my client
code runs two threads with:
Node node =
NodeBuilder.nodeBuilder().client(true).loadConfigSettings(true).node();
Client client = node.client();
System.out.println("got up to here");
client.close();

And I am seeing the "got up to here" messages.
The remote ES server stacks are:

Daemon Thread [elasticsearch[Sofen, Karla]clusterService#updateTask-
pool-5-thread-1] (Suspended (breakpoint at line 39 in
RemoteTransportException))
NodeNotConnectedException(RemoteTransportException).(String,
TransportAddress, String, Throwable) line: 39

NodeNotConnectedException(ConnectTransportException).(DiscoveryNode,
String, String, Throwable) line: 44

NodeNotConnectedException(ConnectTransportException).(DiscoveryNode,
String) line: 32
NodeNotConnectedException.(DiscoveryNode, String) line: 32
NettyTransport.nodeChannel(DiscoveryNode) line: 473
NettyTransport.sendRequest(DiscoveryNode, long, String, Streamable,
TransportRequestOptions) line: 391
TransportService.sendRequest(DiscoveryNode, String, Streamable,
TransportRequestOptions, TransportResponseHandler) line: 182
TransportService.sendRequest(DiscoveryNode, String, Streamable,
TransportResponseHandler) line: 170
PublishClusterStateAction.publish(ClusterState) line: 70
ZenDiscovery.publish(ClusterState) line: 221
DiscoveryService.publish(ClusterState) line: 110
InternalClusterService$2.run() line: 221
ThreadPoolExecutor$Worker.runTask(Runnable) line: 885
ThreadPoolExecutor$Worker.run() line: 907
Thread.run() line: 619

Daemon Thread [elasticsearch[Sofen, Karla]clusterService#updateTask-
pool-5-thread-1] (Suspended (breakpoint at line 58 in
RemoteTransportException))
SendRequestTransportException(RemoteTransportException).fillStack()
line: 58
SendRequestTransportException.fillInStackTrace() line: 34
SendRequestTransportException(RemoteTransportException).fillStack()
line: 58
SendRequestTransportException.fillInStackTrace() line: 34
SendRequestTransportException(Throwable).(String, Throwable)
line: 218
SendRequestTransportException(Exception).(String, Throwable)
line: 59
SendRequestTransportException(RuntimeException).(String,
Throwable) line: 61
SendRequestTransportException(ElasticSearchException).(String,
Throwable) line: 46
SendRequestTransportException(TransportException).(String,
Throwable) line: 34

SendRequestTransportException(RemoteTransportException).(String,
TransportAddress, String, Throwable) line: 39
SendRequestTransportException.(DiscoveryNode, String,
Throwable) line: 30
TransportService.sendRequest(DiscoveryNode, String, Streamable,
TransportRequestOptions, TransportResponseHandler) line: 197
TransportService.sendRequest(DiscoveryNode, String, Streamable,
TransportResponseHandler) line: 170
PublishClusterStateAction.publish(ClusterState) line: 70
ZenDiscovery.publish(ClusterState) line: 221
DiscoveryService.publish(ClusterState) line: 110
InternalClusterService$2.run() line: 221
ThreadPoolExecutor$Worker.runTask(Runnable) line: 885
ThreadPoolExecutor$Worker.run() line: 907
Thread.run() line: 619

I got the first one of them, hit continue a couple of times, and then
got the second one.
You can see the offending lines:
SendRequestTransportException(RemoteTransportException).fillStack()
line: 58
SendRequestTransportException.fillInStackTrace() line: 34

Do you want me to get some value from any variable?

On Nov 6, 9:39 pm, Sebastian sgavar...@gmail.com wrote:

You are right, I thought I needed it, I added a couple of new jars to
my project at the same time and one of them was using slf4j, but not
ES, I removed it and the slf4j messages went away.
Unfortunately not the stackOverflow. I connected the remote debugger
to my ES server, and I see a huge stack of:

SendRequestTransportException(RemoteTransportException.fillStack()
line 58
SendRequestTransportException.fillInStackTrace() line 34
...

Does that tells you something? because for me the chain goes up to
Throwable: public synchronized native Throwable fillInStackTrace();

Sebastian.

On Nov 6, 8:59 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

elasticsearch does not have a dependency on slf4j, are you adding it
just

because of elasticsearch? Just add log4j to the classpath and see if it
happens (without slf4j, any of its jars), or don't add any logging
jars, and

logging will be done using the JDK logging by default.

-shay.banon

On Sun, Nov 7, 2010 at 1:27 AM, Sebastian sgavar...@gmail.com wrote:

Hi Shay,

I tried with open jdk 1.6 and I didn't see the error. Back with
JRockit the exception breakpoint didn't work. Then I did another
simple test, and even though I didn't see the exact same error as
before I think I might have a problem with slf4j.

When trying to run a simple test just joining the cluster from ten
threads:

On Eclipse I see

SLF4J: The following loggers will not work becasue they were created
SLF4J: during the default configuration phase of the underlying
logging system.
SLF4J: See alsohttp://www.slf4j.org/codes.html#substituteLogger
SLF4J: node
SLF4J: node
SLF4J: node
SLF4J: node

On ES server logs I see a couple of:
[2010-11-06 20:23:06,264][INFO ][cluster.service ] [Algrim
the Strong] removed {[Genis-Vell][i0Y0QXoDRbOrDVr2fNFvnw][inet[/
10.0.0.1:5005]]{client=true, data=false},}, reason: zen-disco-
node_failed([Genis-Vell][i0Y0QXoDRbOrDVr2fNFvnw][inet[/10.0.0.1:5005
]]

{client=true, data=false}), reason transport disconnected
Exception in thread "elasticsearch[Algrim the
Strong]clusterService#updateTask-pool-5-thread-14"
java.lang.StackOverflowError
Exception in thread "elasticsearch[Algrim the
Strong]clusterService#updateTask-pool-5-thread-15"
java.lang.StackOverflowError

If I remove slf4j from my poms I see on Eclipse:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Seehttp://www.slf4j.org/codes.html#StaticLoggerBinderfor
further details.
java.lang.NoClassDefFoundError: org/slf4j/impl/StaticLoggerBinder
at
org.slf4j.LoggerFactory.staticInitialize(LoggerFactory.java:83)

   at org.slf4j.LoggerFactory.<clinit>(LoggerFactory.java:73)
   at

org.elasticsearch.common.logging.slf4j.Slf4jESLoggerFactory.newInstance(Slf
4jESLoggerFactory.java:

  1. at

org.elasticsearch.common.logging.ESLoggerFactory.getLogger(ESLoggerFactory.
java:

  1. at
    org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
  1. at
    org.elasticsearch.common.logging.Loggers.getLogger(Loggers.java:
  1. at

org.elasticsearch.node.internal.InternalNode.(InternalNode.java:

  1. at
    org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
   at

org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)

I am not sure if this is related but it could, I never used slf4j
before, but read that by adding:

org.slf4j
slf4j-log4j12
true

to my project it should log to log4j.
Am I messing things up? could you please tell me what's the expected
configuration for a java client api?
Thanks,
Sebastian.

On Nov 6, 6:22 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

An exception breakpoint allows, using the IDE, to break when
specific

exception occurs. In IntelliJ, you go over to the Run Menue, click
the

Breakpoints option, and the second tab there are exception
breakpoints.

You

can simply add one for the StackOverflow exception. Then, the
importnat

bit

here is the debug stack trace, and which methods keep repeating.

If you can try the sun jdk it would be great. Never tried to
debug a

Stack

Overflow in JRockit, but the ones I saw on the sun jdk do show the
stack

trace, and then its easy to understand where it is coming from, as
you

see

the method(s) that keep repeating.

The [tp] thread pool is the main thread pool in elasticsearch.

-shay.banon

On Sat, Nov 6, 2010 at 11:00 PM, Sebastian sgavar...@gmail.com
wrote:

Hi Shay,

...

read more »


(system) #11