# Logstash hangs with OutOfMemoryError **URL:** \ **Category:** Logstash\ **Created:** [October 26, 2015, 5:28pm UTC](https://discuss.elastic.co/t/logstash-hangs-with-outofmemoryerror/33005 "2015-10-26T17:28:32Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![allenmchan](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@allenmchan](https://discuss.elastic.co/u/allenmchan)\ **Post date:** [October 26, 2015, 5:28pm UTC](https://discuss.elastic.co/t/logstash-hangs-with-outofmemoryerror/33005/1 "2015-10-26T17:28:32Z")
Today i noticed that 5/8 logstash indexers were "hung" and i see the below error in /var/log/logstash/logstash.err - Running logstash 1.5.3 with Transport Node for elasticsearch output - Running logstash with LS\_HEAP\_SIZE="2g" in /etc/init.d/logstash config - Running Elasticsearch 1.7.3 - I checked all the elasticsearch data nodes and see no errors. And Marvel says there are no issues with Elasticsearch Heap. (no nodes is \> 70% JVM heap) Has anyone seen this issue before? Oct 26, 2015 9:02:55 AM org.elasticsearch.transport.netty.NettyInternalESLogger warn WARNING: Unexpected exception in the selector loop. java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) at org.elasticsearch.common.netty.channel.socket.nio.SocketReceiveBufferAllocator.newBuffer(SocketReceiveBufferAllocator.java:64) at org.elasticsearch.common.netty.channel.socket.nio.SocketReceiveBufferAllocator.get(SocketReceiveBufferAllocator.java:41) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:62) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 27, 2015, 1:28am UTC](https://discuss.elastic.co/t/logstash-hangs-with-outofmemoryerror/33005/2 "2015-10-27T01:28:00Z")
Is that an ES client node that is OOMing or a Logstash instance? it looks like ES but your notes are a little unclear on that aspect. ---
**Author:** ![allenmchan](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@allenmchan](https://discuss.elastic.co/u/allenmchan)\ **Post date:** [October 27, 2015, 5:04am UTC](https://discuss.elastic.co/t/logstash-hangs-with-outofmemoryerror/33005/3 "2015-10-27T05:04:01Z")
that log came from logstash instance. "i see the below error in /var/log/logstash/logstash.err" ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 27, 2015, 5:35am UTC](https://discuss.elastic.co/t/logstash-hangs-with-outofmemoryerror/33005/4 "2015-10-27T05:35:25Z")
What does your config look like? ---
**Author:** ![allenmchan](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@allenmchan](https://discuss.elastic.co/u/allenmchan)\ **Post date:** [October 27, 2015, 6:02am UTC](https://discuss.elastic.co/t/logstash-hangs-with-outofmemoryerror/33005/5 "2015-10-27T06:02:05Z")
Pretty simple output output { # send logstash metrics to marvel cluster instead of production if "lumberjack\_metric" in [tags] or "syslog\_metric" in [tags] or "redis\_metric" in [tags] { elasticsearch { host =\> ["ip1","ip2"] protocol =\> "http" workers =\> "2" cluster =\> "es\_mon" } } else { elasticsearch { host =\> ["ip4:9350","ip5:9350","ip6:9350"] protocol =\> "transport" cluster =\> "vcc\_cluster" workers =\> "10" } } } ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [October 27, 2015, 2:42pm UTC](https://discuss.elastic.co/t/logstash-hangs-with-outofmemoryerror/33005/6 "2015-10-27T14:42:06Z")
> [@allenmchan](#): > > elasticsearch { > host =\> ["ip4:9350","ip5:9350","ip6:9350"] > protocol =\> "transport" > cluster =\> "vcc\_cluster" > workers =\> "10" > } This quite probably explains why you're experiencing memory issues. You should check how many transport connections are open, but my guess is that it will be in the vicinity of 30. The reason is that with multiple hosts defined it will try to spin up "workers" multiplied by hosts. Since your example is using transport, this would be a java client for each worker spun up, which will consume a considerable amount of memory added up. I recommend switching to using the http protocol for the second example as well. With recent releases of the plugin, it should do some client round-robining and get you the speed you are seeking, without the extreme overhead of 10 java clients per host. ---
**Author:** ![allenmchan](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@allenmchan](https://discuss.elastic.co/u/allenmchan)\ **Post date:** [October 27, 2015, 5:41pm UTC](https://discuss.elastic.co/t/logstash-hangs-with-outofmemoryerror/33005/7 "2015-10-27T17:41:09Z")
Thats for the suggestion Aaron. I will move to HTTP protocol since 2.0 release will default to that anyways. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:25am UTC](https://discuss.elastic.co/t/logstash-hangs-with-outofmemoryerror/33005/8 "2017-07-06T05:25:12Z")