# Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException: (TimeoutError) watchdog timeout

**URL:** <https://discuss.elastic.co/t/exception-in-thread-logstash-runner-org-jruby-exceptions-raiseexception-timeouterror-watchdog-timeout/13386>\
**Category:** Elasticsearch\
**Created:** [August 29, 2013, 8:00am UTC](https://discuss.elastic.co/t/exception-in-thread-logstash-runner-org-jruby-exceptions-raiseexception-timeouterror-watchdog-timeout/13386 "2013-08-29T08:00:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![haim\_ari](https://avatars.discourse-cdn.com/v4/letter/h/d9b06d/32.png) [@haim\_ari](https://discuss.elastic.co/u/haim_ari)\
**Post date:** [August 29, 2013, 8:00am UTC](https://discuss.elastic.co/t/exception-in-thread-logstash-runner-org-jruby-exceptions-raiseexception-timeouterror-watchdog-timeout/13386/1 "2013-08-29T08:00:51Z")

</div>

PowerEdge R420  
Physical Memory: 24GB  
2 X Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz (Total 24 cores with  
Multi-threading)  
Centos \_6.4\_x86\_64

_Logstash error:_

Exception in thread "LogStash::Runner" org.jruby.exceptions.RaiseException:  
(TimeoutError) watchdog timeout

seems to be caused by:

_ElasticSearch Error:_

[2013-08-29 00:04:07,236][WARN  
][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the  
selector loop.  
java.lang.OutOfMemoryError: Java heap space  
at  
org.elasticsearch.common.netty.buffer.HeapChannelBuffer.(HeapChannelBuffer.java:42)  
at  
org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer.(BigEndianHeapChannelBuffer.java:34)  
at  
org.elasticsearch.common.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)  
at  
org.elasticsearch.common.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:68)  
at  
org.elasticsearch.common.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChannelBufferFactory.java:48)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)  
at  
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
at  
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
at java.lang.Thread.run(Thread.java:724)  
[2013-08-29 00:08:33,743][WARN][index.engine.robin]  
[elastic-master] [application-2013.08.29][0] failed engine  
java.lang.OutOfMemoryError: Java heap space  
[2013-08-29 00:08:33,743][WARN][transport.netty]  
[elastic-master] exception caught on transport layer [[id: 0x7e825467,  
/192.168.x.x:33673 =\> /192.168.x.x:9300]], closing connection

_Logstash runs with:_  
-Xmx8192M -Xms2048M -XX:+UseConcMarkSweepGC

_ElasticSearch runs with:_  
-Xms4g -Xmx8g -Xss512k -Djava.awt.headless=true -XX:+UseParNewGC  
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75  
-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCondCardMark  
-XX:+HeapDumpOnOutOfMemoryError -Delasticsearch  
-Des.path.home=/opt/elasticsearch-0.20.6 -cp  
:/opt/elasticsearch-0.20.6/lib/elasticsearch-0.20.6.jar:/opt/elasticsearch-0.20.6/lib/_:/opt/elasticsearch-0.20.6/lib/sigar/_  
org.elasticsearch.bootstrap.ElasticSearch

_Logstash config:_

input {

```
    gelf {
            type => application
            port => 12201
    }

    gelf {
            type => access_log
            port => 12202
            format => 'plain'
    }

    gelf {
            type => trackinstall
            port => 12203
    }

```

}

filter {

```
grok {
    type => "access_log"
    patterns_dir => "./patterns"
    pattern => "%{IP:client} \- \- \"%{WORD:method} 

```

%{URIPATH:uri\_path}%{URIPARAM:params} %{DATA:protocol}" %{NUMBER:code}  
%{NUMBER:bytes}"  
pattern =\> "%{IP:client} - - "%{WORD:method}  
%{URIPATH:uri\_path}%{URIPARAM:params} %{DATA:protocol}" %{NUMBER:code} -"  
pattern =\> "%{IP:client} - - "%{WORD:method} %{URIPATH:uri\_path}  
%{DATA:protocol}" %{NUMBER:code} %{NUMBER:bytes}"  
}

```
grok {
    type => "trackinstall"
    patterns_dir => "./patterns"
    pattern => "%{IP:client} \- \- \"%{WORD:method} 

```

%{URIPATH:uri\_path}%{URIPARAM:params} %{DATA:protocol}" %{NUMBER:code}  
%{NUMBER:bytes}"  
pattern =\> "%{IP:client} - - "%{WORD:method}  
%{URIPATH:uri\_path}%{URIPARAM:params} %{DATA:protocol}" %{NUMBER:code} -"  
pattern =\> "%{IP:client} - - "%{WORD:method} %{URIPATH:uri\_path}  
%{DATA:protocol}" %{NUMBER:code} %{NUMBER:bytes}"  
}

```
kv {
    type => "access_log"
    fields => ["params"]
    field_split=> "&? "
}

kv {
    type => "trackinstall"
    fields => ["params"]
    field_split=> "&? "
}

urldecode {
        type => "access_log"
        all_fields => true
}
urldecode {
        type => "trackinstall"
        all_fields => true
}

```

}

output {  
elasticsearch {  
host =\> "192.168.x.x"  
bind\_host =\> "192.168.x.x"  
cluster =\> "startapp"  
node\_name =\> "logstash\_node\_ls1"  
max\_inflight\_requests =\> 250000  
type =\> "application"  
index =\> "application-%{+YYYY.MM.dd}"  
}  
elasticsearch {  
host =\> "192.168.x.x"  
bind\_host =\> "192.168.x.x"  
cluster =\> "startapp"  
node\_name =\> "logstash\_node\_ls1"  
max\_inflight\_requests =\> 250000  
type =\> "access\_log"  
index =\> "access-%{+YYYY.MM.dd}"  
}  
elasticsearch {  
host =\> "192.168.x.x"  
bind\_host =\> "192.168.x.x"  
cluster =\> "startapp"  
node\_name =\> "logstash\_node\_ls1"  
max\_inflight\_requests =\> 250000  
type =\> "trackinstall"  
index =\> "trackinstall-%{+YYYY.MM.dd}"  
}

}

_ElasticSearch config:_  
\*  
\*  
cluster.name: somename

node.name: "elastic-master"

index.number\_of\_shards: 1  
index.number\_of\_replicas: 0

bootstrap.mlockall: true

network.host: 192.168.x.x

gateway.expected\_nodes: 1

indices.memory.index\_buffer\_size: 25%  
index.refresh\_interval: 30  
index.translog.flush\_threshold\_ops: 5000  
index.store.compress.stored: true

threadpool.search.type: fixed  
threadpool.search.size: 20  
threadpool.search.queue\_size: 100

threadpool.index.type: fixed  
threadpool.index.size: 60  
indices.store.throttle.type: none  
indices.store.throttle.max\_bytes\_per\_sec: 5m

index.cache.field.max\_size: 5000  
index.cache.field.expire: 10m

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![haim\_ari](https://avatars.discourse-cdn.com/v4/letter/h/d9b06d/32.png) [@haim\_ari](https://discuss.elastic.co/u/haim_ari)\
**Post date:** [August 29, 2013, 8:02am UTC](https://discuss.elastic.co/t/exception-in-thread-logstash-runner-org-jruby-exceptions-raiseexception-timeouterror-watchdog-timeout/13386/2 "2013-08-29T08:02:04Z")

</div>

_update:_  
\*  
\*  
_I just restarted elasticsearch and after just a few minutes got:_

[2013-08-29 07:37:10,590][WARN  
][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the  
selector loop.  
java.lang.OutOfMemoryError: Java heap space  
at  
org.elasticsearch.common.netty.buffer.HeapChannelBuffer.(HeapChannelBuffer.java:42)  
at  
org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer.(BigEndianHeapChannelBuffer.java:34)  
at  
org.elasticsearch.common.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)  
at  
org.elasticsearch.common.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:68)  
at  
org.elasticsearch.common.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChannelBufferFactory.java:48)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)  
at  
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
at  
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
at java.lang.Thread.run(Thread.java:724)  
[2013-08-29 07:37:10,590][WARN][transport.netty]  
[elastic-master] exception caught on transport layer [[id: 0x4bc8ef0f,  
/192.168.251.19:52908 =\> /192.168.251.19:9300]], closing connection  
java.lang.OutOfMemoryError: Java heap space  
[2013-08-29 07:37:10,616][WARN][transport.netty]  
[elastic-master] exception caught on transport layer [[id: 0x4bc8ef0f,  
/192.168.251.19:52908 :\> /192.168.251.19:9300]], closing connection

On Thursday, August 29, 2013 11:00:51 AM UTC+3, imneo wrote:

> PowerEdge R420  
> Physical Memory: 24GB  
> 2 X Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz (Total 24 cores with  
> Multi-threading)  
> Centos \_6.4\_x86\_64
> 
> _Logstash error:_
> 
> Exception in thread "LogStash::Runner"  
> org.jruby.exceptions.RaiseException: (TimeoutError) watchdog timeout
> 
> seems to be caused by:
> 
> _Elasticsearch Error:_
> 
> [2013-08-29 00:04:07,236][WARN  
> ][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the  
> selector loop.  
> java.lang.OutOfMemoryError: Java heap space  
> at  
> org.elasticsearch.common.netty.buffer.HeapChannelBuffer.(HeapChannelBuffer.java:42)  
> at  
> org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer.(BigEndianHeapChannelBuffer.java:34)  
> at  
> org.elasticsearch.common.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)  
> at  
> org.elasticsearch.common.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:68)  
> at  
> org.elasticsearch.common.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChannelBufferFactory.java:48)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)  
> at  
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at  
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)  
> at  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> at java.lang.Thread.run(Thread.java:724)  
> [2013-08-29 00:08:33,743][WARN][index.engine.robin]  
> [elastic-master] [application-2013.08.29][0] failed engine  
> java.lang.OutOfMemoryError: Java heap space  
> [2013-08-29 00:08:33,743][WARN][transport.netty]  
> [elastic-master] exception caught on transport layer [[id: 0x7e825467,  
> /192.168.x.x:33673 =\> /192.168.x.x:9300]], closing connection
> 
> _Logstash runs with:_  
> -Xmx8192M -Xms2048M -XX:+UseConcMarkSweepGC
> 
> _Elasticsearch runs with:_  
> -Xms4g -Xmx8g -Xss512k -Djava.awt.headless=true -XX:+UseParNewGC  
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75  
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCondCardMark  
> -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch  
> -Des.path.home=/opt/elasticsearch-0.20.6 -cp  
> :/opt/elasticsearch-0.20.6/lib/elasticsearch-0.20.6.jar:/opt/elasticsearch-0.20.6/lib/_:/opt/elasticsearch-0.20.6/lib/sigar/_  
> org.elasticsearch.bootstrap.Elasticsearch
> 
> _Logstash config:_
> 
> input {
> 
> ```
> gelf {
> type => application
> port => 12201
> }
> 
> gelf {
> type => access_log
> port => 12202
> format => 'plain'
> }
> 
> gelf {
> type => trackinstall
> port => 12203
> }
> 
> ```
> 
> }
> 
> filter {
> 
> ```
> grok {
> type => "access_log"
> patterns_dir => "./patterns"
> pattern => "%{IP:client} \- \- \"%{WORD:method} 
> 
> ```
> 
> %{URIPATH:uri\_path}%{URIPARAM:params} %{DATA:protocol}" %{NUMBER:code}  
> %{NUMBER:bytes}"  
> pattern =\> "%{IP:client} - - "%{WORD:method}  
> %{URIPATH:uri\_path}%{URIPARAM:params} %{DATA:protocol}" %{NUMBER:code} -"  
> pattern =\> "%{IP:client} - - "%{WORD:method}  
> %{URIPATH:uri\_path} %{DATA:protocol}" %{NUMBER:code} %{NUMBER:bytes}"  
> }
> 
> ```
> grok {
> type => "trackinstall"
> patterns_dir => "./patterns"
> pattern => "%{IP:client} \- \- \"%{WORD:method} 
> 
> ```
> 
> %{URIPATH:uri\_path}%{URIPARAM:params} %{DATA:protocol}" %{NUMBER:code}  
> %{NUMBER:bytes}"  
> pattern =\> "%{IP:client} - - "%{WORD:method}  
> %{URIPATH:uri\_path}%{URIPARAM:params} %{DATA:protocol}" %{NUMBER:code} -"  
> pattern =\> "%{IP:client} - - "%{WORD:method}  
> %{URIPATH:uri\_path} %{DATA:protocol}" %{NUMBER:code} %{NUMBER:bytes}"  
> }
> 
> ```
> kv {
> type => "access_log"
> fields => ["params"]
> field_split=> "&? "
> }
> 
> kv {
> type => "trackinstall"
> fields => ["params"]
> field_split=> "&? "
> }
> 
> urldecode {
> type => "access_log"
> all_fields => true
> }
> urldecode {
> type => "trackinstall"
> all_fields => true
> }
> 
> ```
> 
> }
> 
> output {  
> elasticsearch {  
> host =\> "192.168.x.x"  
> bind\_host =\> "192.168.x.x"  
> cluster =\> "startapp"  
> node\_name =\> "logstash\_node\_ls1"  
> max\_inflight\_requests =\> 250000  
> type =\> "application"  
> index =\> "application-%{+YYYY.MM.dd}"  
> }  
> elasticsearch {  
> host =\> "192.168.x.x"  
> bind\_host =\> "192.168.x.x"  
> cluster =\> "startapp"  
> node\_name =\> "logstash\_node\_ls1"  
> max\_inflight\_requests =\> 250000  
> type =\> "access\_log"  
> index =\> "access-%{+YYYY.MM.dd}"  
> }  
> elasticsearch {  
> host =\> "192.168.x.x"  
> bind\_host =\> "192.168.x.x"  
> cluster =\> "startapp"  
> node\_name =\> "logstash\_node\_ls1"  
> max\_inflight\_requests =\> 250000  
> type =\> "trackinstall"  
> index =\> "trackinstall-%{+YYYY.MM.dd}"  
> }
> 
> }
> 
> _Elasticsearch config:_  
> \*  
> \*  
> cluster.name: somename
> 
> node.name: "elastic-master"
> 
> index.number\_of\_shards: 1  
> index.number\_of\_replicas: 0
> 
> bootstrap.mlockall: true
> 
> network.host: 192.168.x.x
> 
> gateway.expected\_nodes: 1
> 
> indices.memory.index\_buffer\_size: 25%  
> index.refresh\_interval: 30  
> index.translog.flush\_threshold\_ops: 5000  
> index.store.compress.stored: true
> 
> threadpool.search.type: fixed  
> threadpool.search.size: 20  
> threadpool.search.queue\_size: 100
> 
> threadpool.index.type: fixed  
> threadpool.index.size: 60  
> indices.store.throttle.type: none  
> indices.store.throttle.max\_bytes\_per\_sec: 5m
> 
> index.cache.field.max\_size: 5000  
> index.cache.field.expire: 10m

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![haim\_ari](https://avatars.discourse-cdn.com/v4/letter/h/d9b06d/32.png) [@haim\_ari](https://discuss.elastic.co/u/haim_ari)\
**Post date:** [August 29, 2013, 8:05am UTC](https://discuss.elastic.co/t/exception-in-thread-logstash-runner-org-jruby-exceptions-raiseexception-timeouterror-watchdog-timeout/13386/3 "2013-08-29T08:05:13Z")

</div>

_update:_  
\*  
\*  
_I just restarted elasticsearch and after just a few minutes got:_

[2013-08-29 07:37:10,590][WARN  
][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the  
selector loop.  
java.lang.OutOfMemoryError: Java heap space  
at  
org.elasticsearch.common.netty.buffer.HeapChannelBuffer.(HeapChannelBuffer.java:42)  
at  
org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer.(BigEndianHeapChannelBuffer.java:34)  
at  
org.elasticsearch.common.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)  
at  
org.elasticsearch.common.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:68)  
at  
org.elasticsearch.common.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChannelBufferFactory.java:48)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)  
at  
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)  
at  
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)  
at  
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
at  
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
at java.lang.Thread.run(Thread.java:724)  
[2013-08-29 07:37:10,590][WARN][transport.netty]  
[elastic-master] exception caught on transport layer [[id: 0x4bc8ef0f, /  
192.168.x.x:52908 [http://192.168.251.19:52908/](http://192.168.251.19:52908/) =\> /192.168.x.x:9300]],  
closing connection  
java.lang.OutOfMemoryError: Java heap space  
[2013-08-29 07:37:10,616][WARN][transport.netty]  
[elastic-master] exception caught on transport layer [[id: 0x4bc8ef0f, /  
192.168.x.x:52908 [http://192.168.251.19:52908/](http://192.168.251.19:52908/) :\> /192.168.x.x:9300]],  
closing connection

On Thursday, August 29, 2013 11:00:51 AM UTC+3, imneo wrote:

> PowerEdge R420  
> Physical Memory: 24GB  
> 2 X Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz (Total 24 cores with  
> Multi-threading)  
> Centos \_6.4\_x86\_64
> 
> _Logstash error:_
> 
> Exception in thread "LogStash::Runner"  
> org.jruby.exceptions.RaiseException: (TimeoutError) watchdog timeout
> 
> seems to be caused by:
> 
> _Elasticsearch Error:_
> 
> [2013-08-29 00:04:07,236][WARN  
> ][netty.channel.socket.nio.AbstractNioSelector] Unexpected exception in the  
> selector loop.  
> java.lang.OutOfMemoryError: Java heap space  
> at  
> org.elasticsearch.common.netty.buffer.HeapChannelBuffer.(HeapChannelBuffer.java:42)  
> at  
> org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer.(BigEndianHeapChannelBuffer.java:34)  
> at  
> org.elasticsearch.common.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)  
> at  
> org.elasticsearch.common.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:68)  
> at  
> org.elasticsearch.common.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChannelBufferFactory.java:48)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:80)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)  
> at  
> org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)  
> at  
> org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)  
> at  
> org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)  
> at  
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)  
> at java.lang.Thread.run(Thread.java:724)  
> [2013-08-29 00:08:33,743][WARN][index.engine.robin]  
> [elastic-master] [application-2013.08.29][0] failed engine  
> java.lang.OutOfMemoryError: Java heap space  
> [2013-08-29 00:08:33,743][WARN][transport.netty]  
> [elastic-master] exception caught on transport layer [[id: 0x7e825467,  
> /192.168.x.x:33673 =\> /192.168.x.x:9300]], closing connection
> 
> _Logstash runs with:_  
> -Xmx8192M -Xms2048M -XX:+UseConcMarkSweepGC
> 
> _Elasticsearch runs with:_  
> -Xms4g -Xmx8g -Xss512k -Djava.awt.headless=true -XX:+UseParNewGC  
> -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75  
> -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCondCardMark  
> -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch  
> -Des.path.home=/opt/elasticsearch-0.20.6 -cp  
> :/opt/elasticsearch-0.20.6/lib/elasticsearch-0.20.6.jar:/opt/elasticsearch-0.20.6/lib/_:/opt/elasticsearch-0.20.6/lib/sigar/_  
> org.elasticsearch.bootstrap.Elasticsearch
> 
> _Logstash config:_
> 
> input {
> 
> ```
> gelf {
> type => application
> port => 12201
> }
> 
> gelf {
> type => access_log
> port => 12202
> format => 'plain'
> }
> 
> gelf {
> type => trackinstall
> port => 12203
> }
> 
> ```
> 
> }
> 
> filter {
> 
> ```
> grok {
> type => "access_log"
> patterns_dir => "./patterns"
> pattern => "%{IP:client} \- \- \"%{WORD:method} 
> 
> ```
> 
> %{URIPATH:uri\_path}%{URIPARAM:params} %{DATA:protocol}" %{NUMBER:code}  
> %{NUMBER:bytes}"  
> pattern =\> "%{IP:client} - - "%{WORD:method}  
> %{URIPATH:uri\_path}%{URIPARAM:params} %{DATA:protocol}" %{NUMBER:code} -"  
> pattern =\> "%{IP:client} - - "%{WORD:method}  
> %{URIPATH:uri\_path} %{DATA:protocol}" %{NUMBER:code} %{NUMBER:bytes}"  
> }
> 
> ```
> grok {
> type => "trackinstall"
> patterns_dir => "./patterns"
> pattern => "%{IP:client} \- \- \"%{WORD:method} 
> 
> ```
> 
> %{URIPATH:uri\_path}%{URIPARAM:params} %{DATA:protocol}" %{NUMBER:code}  
> %{NUMBER:bytes}"  
> pattern =\> "%{IP:client} - - "%{WORD:method}  
> %{URIPATH:uri\_path}%{URIPARAM:params} %{DATA:protocol}" %{NUMBER:code} -"  
> pattern =\> "%{IP:client} - - "%{WORD:method}  
> %{URIPATH:uri\_path} %{DATA:protocol}" %{NUMBER:code} %{NUMBER:bytes}"  
> }
> 
> ```
> kv {
> type => "access_log"
> fields => ["params"]
> field_split=> "&? "
> }
> 
> kv {
> type => "trackinstall"
> fields => ["params"]
> field_split=> "&? "
> }
> 
> urldecode {
> type => "access_log"
> all_fields => true
> }
> urldecode {
> type => "trackinstall"
> all_fields => true
> }
> 
> ```
> 
> }
> 
> output {  
> elasticsearch {  
> host =\> "192.168.x.x"  
> bind\_host =\> "192.168.x.x"  
> cluster =\> "startapp"  
> node\_name =\> "logstash\_node\_ls1"  
> max\_inflight\_requests =\> 250000  
> type =\> "application"  
> index =\> "application-%{+YYYY.MM.dd}"  
> }  
> elasticsearch {  
> host =\> "192.168.x.x"  
> bind\_host =\> "192.168.x.x"  
> cluster =\> "startapp"  
> node\_name =\> "logstash\_node\_ls1"  
> max\_inflight\_requests =\> 250000  
> type =\> "access\_log"  
> index =\> "access-%{+YYYY.MM.dd}"  
> }  
> elasticsearch {  
> host =\> "192.168.x.x"  
> bind\_host =\> "192.168.x.x"  
> cluster =\> "startapp"  
> node\_name =\> "logstash\_node\_ls1"  
> max\_inflight\_requests =\> 250000  
> type =\> "trackinstall"  
> index =\> "trackinstall-%{+YYYY.MM.dd}"  
> }
> 
> }
> 
> _Elasticsearch config:_  
> \*  
> \*  
> cluster.name: somename
> 
> node.name: "elastic-master"
> 
> index.number\_of\_shards: 1  
> index.number\_of\_replicas: 0
> 
> bootstrap.mlockall: true
> 
> network.host: 192.168.x.x
> 
> gateway.expected\_nodes: 1
> 
> indices.memory.index\_buffer\_size: 25%  
> index.refresh\_interval: 30  
> index.translog.flush\_threshold\_ops: 5000  
> index.store.compress.stored: true
> 
> threadpool.search.type: fixed  
> threadpool.search.size: 20  
> threadpool.search.queue\_size: 100
> 
> threadpool.index.type: fixed  
> threadpool.index.size: 60  
> indices.store.throttle.type: none  
> indices.store.throttle.max\_bytes\_per\_sec: 5m
> 
> index.cache.field.max\_size: 5000  
> index.cache.field.expire: 10m

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [August 29, 2013, 9:31pm UTC](https://discuss.elastic.co/t/exception-in-thread-logstash-runner-org-jruby-exceptions-raiseexception-timeouterror-watchdog-timeout/13386/4 "2013-08-29T21:31:08Z")

</div>

The main thing I see from the log output is "java.lang.OutOfMemoryError:  
Java heap space". Quite simply, you need more memory. The memory usage is  
greatly improved in elasticsearch 0.90.x and I would suggest upgrading if  
possible. If not, you need to debug your queries and see how to minimize  
memory issues.

Cheers,

Ivan

On Thu, Aug 29, 2013 at 1:05 AM, imneo [haimari1@gmail.com](mailto:haimari1@gmail.com) wrote:

> _update:_  
> \*  
> \*  
> _I just restarted elasticsearch and after just a few minutes got:_
> 
> [2013-08-29 07:37:10,590][WARN][netty.channel.socket.nio.\*\*AbstractNioSelector]  
> Unexpected exception in the selector loop.  
> java.lang.OutOfMemoryError: Java heap space  
> at org.elasticsearch.common. **netty.buffer.**  
> HeapChannelBuffer.(\*\*HeapChannelBuffer.java:42)  
> at org.elasticsearch.common. **netty.buffer.**  
> BigEndianHeapChannelBuffer.\<\*\*init\>( **BigEndianHeapChannelBuffer.**  
> java:34)  
> at org.elasticsearch.common. **netty.buffer.ChannelBuffers.**  
> buffer(ChannelBuffers.java:\*\*134)  
> at org.elasticsearch.common. **netty.buffer.**  
> HeapChannelBufferFactory.\*\*getBuffer(\*\*HeapChannelBufferFactory.java:\*\*68)  
> at org.elasticsearch.common. **netty.buffer.**  
> AbstractChannelBufferFactory.\*\*getBuffer( **AbstractChannelBufferFactory.**  
> java:48)  
> at org.elasticsearch.common. **netty.channel.socket.nio.**  
> NioWorker.read(NioWorker.java:\*\*80)  
> at org.elasticsearch.common. **netty.channel.socket.nio.**  
> AbstractNioWorker.process(\*\*AbstractNioWorker.java:107)  
> at org.elasticsearch.common. **netty.channel.socket.nio.**  
> AbstractNioSelector.run(\*\*AbstractNioSelector.java:312)  
> at org.elasticsearch.common. **netty.channel.socket.nio.**  
> AbstractNioWorker.run(\*\*AbstractNioWorker.java:88)  
> at org.elasticsearch.common. **netty.channel.socket.nio.**  
> NioWorker.run(NioWorker.java:\*\*178)  
> at org.elasticsearch.common. **netty.util.**  
> ThreadRenamingRunnable.run(\*\*ThreadRenamingRunnable.java:\*\*108)  
> at org.elasticsearch.common. **netty.util.internal.**  
> DeadLockProofWorker$1.run(\*\*DeadLockProofWorker.java:42)  
> at java.util.concurrent.**ThreadPoolExecutor.runWorker(**  
> ThreadPoolExecutor.java:1145)  
> at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**  
> ThreadPoolExecutor.java:615)  
> at java.lang.Thread.run(Thread.\*\*java:724)  
> [2013-08-29 07:37:10,590][WARN][transport.netty]  
> [elastic-master] exception caught on transport layer [[id: 0x4bc8ef0f, /  
> 192.168.x.x:52908 [http://192.168.251.19:52908/](http://192.168.251.19:52908/) =\> /192.168.x.x:9300]],  
> closing connection  
> java.lang.OutOfMemoryError: Java heap space  
> [2013-08-29 07:37:10,616][WARN][transport.netty]  
> [elastic-master] exception caught on transport layer [[id: 0x4bc8ef0f, /  
> 192.168.x.x:52908 [http://192.168.251.19:52908/](http://192.168.251.19:52908/) :\> /192.168.x.x:9300]],  
> closing connection
> 
> On Thursday, August 29, 2013 11:00:51 AM UTC+3, imneo wrote:
> 
> > PowerEdge R420  
> > Physical Memory: 24GB  
> > 2 X Intel(R) Xeon(R) CPU E5-2420 0 @ 1.90GHz (Total 24 cores with  
> > Multi-threading)  
> > Centos \_6.4\_x86\_64
> > 
> > _Logstash error:_
> > 
> > Exception in thread "LogStash::Runner" org.jruby.exceptions.\*\*RaiseException:  
> > (TimeoutError) watchdog timeout
> > 
> > seems to be caused by:
> > 
> > _Elasticsearch Error:_
> > 
> > [2013-08-29 00:04:07,236][WARN][netty.channel.socket.nio.\*\*AbstractNioSelector]  
> > Unexpected exception in the selector loop.  
> > java.lang.OutOfMemoryError: Java heap space  
> > at org.elasticsearch.common. **netty.buffer.**  
> > HeapChannelBuffer.(\*\*HeapChannelBuffer.java:42)  
> > at org.elasticsearch.common. **netty.buffer.**  
> > BigEndianHeapChannelBuffer.\<\*\*init\>( **BigEndianHeapChannelBuffer.**  
> > java:34)  
> > at org.elasticsearch.common. **netty.buffer.ChannelBuffers.**  
> > buffer(ChannelBuffers.java:\*\*134)  
> > at org.elasticsearch.common. **netty.buffer.**  
> > HeapChannelBufferFactory.\*\*getBuffer( **HeapChannelBufferFactory.java:**  
> > 68)  
> > at org.elasticsearch.common. **netty.buffer.**  
> > AbstractChannelBufferFactory.\*\*getBuffer(\*_AbstractChannelBufferFactory._  
> > \*java:48)  
> > at org.elasticsearch.common. **netty.channel.socket.nio.**  
> > NioWorker.read(NioWorker.java:\*\*80)  
> > at org.elasticsearch.common. **netty.channel.socket.nio.**  
> > AbstractNioWorker.process(\*\*AbstractNioWorker.java:107)  
> > at org.elasticsearch.common. **netty.channel.socket.nio.**  
> > AbstractNioSelector.run(\*\*AbstractNioSelector.java:312)  
> > at org.elasticsearch.common. **netty.channel.socket.nio.**  
> > AbstractNioWorker.run(\*\*AbstractNioWorker.java:88)  
> > at org.elasticsearch.common. **netty.channel.socket.nio.**  
> > NioWorker.run(NioWorker.java:\*\*178)  
> > at org.elasticsearch.common. **netty.util.**  
> > ThreadRenamingRunnable.run(\*\*ThreadRenamingRunnable.java:\*\*108)  
> > at org.elasticsearch.common. **netty.util.internal.**  
> > DeadLockProofWorker$1.run(\*\*DeadLockProofWorker.java:42)  
> > at java.util.concurrent.**ThreadPoolExecutor.runWorker(**  
> > ThreadPoolExecutor.java:1145)  
> > at java.util.concurrent.**ThreadPoolExecutor$Worker.run(**  
> > ThreadPoolExecutor.java:615)  
> > at java.lang.Thread.run(Thread.\*\*java:724)  
> > [2013-08-29 00:08:33,743][WARN][index.engine.robin]  
> > [elastic-master] [application-2013.08.29][0] failed engine  
> > java.lang.OutOfMemoryError: Java heap space  
> > [2013-08-29 00:08:33,743][WARN][transport.netty]  
> > [elastic-master] exception caught on transport layer [[id: 0x7e825467,  
> > /192.168.x.x:33673 =\> /192.168.x.x:9300]], closing connection
> > 
> > _Logstash runs with:_  
> > -Xmx8192M -Xms2048M -XX:+UseConcMarkSweepGC
> > 
> > _Elasticsearch runs with:_  
> > -Xms4g -Xmx8g -Xss512k -Djava.awt.headless=true -XX:+UseParNewGC  
> > -XX:+UseConcMarkSweepGC -XX: **CMSInitiatingOccupancyFraction** =75 -XX:+\*\*UseCMSInitiatingOccupancyOnly  
> > -XX:+UseCondCardMark -XX:+\*_HeapDumpOnOutOfMemoryError -Delasticsearch  
> > -Des.path.home=/opt/**elasticsearch-0.20.6 -cp  
> > :/opt/elasticsearch-0.20.6/lib/elasticsearch-0.20.6.jar:/  
> > opt/elasticsearch-0.20.6/lib/**_:/opt/elasticsearch-0.20.6/\*_lib/sigar/_  
> > org.elasticsearch.bootstrap.\*\*Elasticsearch
> > 
> > _Logstash config:_
> > 
> > input {
> > 
> > ```
> > gelf {
> > type => application
> > port => 12201
> > }
> > 
> > gelf {
> > type => access_log
> > port => 12202
> > format => 'plain'
> > }
> > 
> > gelf {
> > type => trackinstall
> > port => 12203
> > }
> > 
> > ```
> > 
> > }
> > 
> > filter {
> > 
> > ```
> > grok {
> > type => "access_log"
> > patterns_dir => "./patterns"
> > pattern => "%{IP:client} \- \- \"%{WORD:method}
> > 
> > ```
> > 
> > %{URIPATH:uri\_path}%{URIPARAM:\*\*params} %{DATA:protocol}"  
> > %{NUMBER:code} %{NUMBER:bytes}"  
> > pattern =\> "%{IP:client} - - "%{WORD:method}  
> > %{URIPATH:uri\_path}%{URIPARAM:\*\*params} %{DATA:protocol}"  
> > %{NUMBER:code} -"  
> > pattern =\> "%{IP:client} - - "%{WORD:method}  
> > %{URIPATH:uri\_path} %{DATA:protocol}" %{NUMBER:code} %{NUMBER:bytes}"  
> > }
> > 
> > ```
> > grok {
> > type => "trackinstall"
> > patterns_dir => "./patterns"
> > pattern => "%{IP:client} \- \- \"%{WORD:method}
> > 
> > ```
> > 
> > %{URIPATH:uri\_path}%{URIPARAM:\*\*params} %{DATA:protocol}"  
> > %{NUMBER:code} %{NUMBER:bytes}"  
> > pattern =\> "%{IP:client} - - "%{WORD:method}  
> > %{URIPATH:uri\_path}%{URIPARAM:\*\*params} %{DATA:protocol}"  
> > %{NUMBER:code} -"  
> > pattern =\> "%{IP:client} - - "%{WORD:method}  
> > %{URIPATH:uri\_path} %{DATA:protocol}" %{NUMBER:code} %{NUMBER:bytes}"  
> > }
> > 
> > ```
> > kv {
> > type => "access_log"
> > fields => ["params"]
> > field_split=> "&? "
> > }
> > 
> > kv {
> > type => "trackinstall"
> > fields => ["params"]
> > field_split=> "&? "
> > }
> > 
> > urldecode {
> > type => "access_log"
> > all_fields => true
> > }
> > urldecode {
> > type => "trackinstall"
> > all_fields => true
> > }
> > 
> > ```
> > 
> > }
> > 
> > output {  
> > elasticsearch {  
> > host =\> "192.168.x.x"  
> > bind\_host =\> "192.168.x.x"  
> > cluster =\> "startapp"  
> > node\_name =\> "logstash\_node\_ls1"  
> > max\_inflight\_requests =\> 250000  
> > type =\> "application"  
> > index =\> "application-%{+YYYY.MM.dd}"  
> > }  
> > elasticsearch {  
> > host =\> "192.168.x.x"  
> > bind\_host =\> "192.168.x.x"  
> > cluster =\> "startapp"  
> > node\_name =\> "logstash\_node\_ls1"  
> > max\_inflight\_requests =\> 250000  
> > type =\> "access\_log"  
> > index =\> "access-%{+YYYY.MM.dd}"  
> > }  
> > elasticsearch {  
> > host =\> "192.168.x.x"  
> > bind\_host =\> "192.168.x.x"  
> > cluster =\> "startapp"  
> > node\_name =\> "logstash\_node\_ls1"  
> > max\_inflight\_requests =\> 250000  
> > type =\> "trackinstall"  
> > index =\> "trackinstall-%{+YYYY.MM.dd}"  
> > }
> > 
> > }
> > 
> > _Elasticsearch config:_  
> > \*  
> > \*  
> > cluster.name: somename
> > 
> > node.name: "elastic-master"
> > 
> > index.number\_of\_shards: 1  
> > index.number\_of\_replicas: 0
> > 
> > bootstrap.mlockall: true
> > 
> > network.host: 192.168.x.x
> > 
> > gateway.expected\_nodes: 1
> > 
> > indices.memory.index\_buffer\_\*\*size: 25%  
> > index.refresh\_interval: 30  
> > index.translog.flush\_\*\*threshold\_ops: 5000  
> > index.store.compress.stored: true
> > 
> > threadpool.search.type: fixed  
> > threadpool.search.size: 20  
> > threadpool.search.queue\_size: 100
> > 
> > threadpool.index.type: fixed  
> > threadpool.index.size: 60  
> > indices.store.throttle.type: none  
> > indices.store.throttle.max\_\*\*bytes\_per\_sec: 5m
> > 
> > index.cache.field.max\_size: 5000  
> > index.cache.field.expire: 10m
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:18am UTC](https://discuss.elastic.co/t/exception-in-thread-logstash-runner-org-jruby-exceptions-raiseexception-timeouterror-watchdog-timeout/13386/5 "2017-07-06T02:18:53Z")

</div>