ElasticsearchIllegalStateException: Maximum number of bytes allocated exceeded

stang07 · June 16, 2014, 9:18pm

I am getting this error constantly and the 5 nodes in the cluster goes down
one by one. Is there a way to fix the issue? Could it be a corrupt data
issue?

[2014-06-16 13:13:34,422][WARN ][discovery.zen.ping.multicast] [nw-es-201]
failed to receive confirmation on sent ping response to
[[nw-es-202][xOi0MSQBQ4OoQjpuIDUjtA][nw-es-202.prod.pcln.com][inet[/172.21.88.113:9320]]{master=true}]
org.elasticsearch.transport.SendRequestTransportException:
[nw-es-202][inet[/172.21.88.113:9320]][discovery/zen/multicast]
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:219)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:190)
at
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.handleNodePingRequest(MulticastZenPing.java:462)
at
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.onMessage(MulticastZenPing.java:334)
at
org.elasticsearch.common.network.MulticastChannel$Plain$Receiver.run(MulticastChannel.java:363)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Maximum
number of bytes allocated exceeded: [836334432] (> 836318003)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:384)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:398)
at
org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:423)
at
org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:431)
at
org.elasticsearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:60)
at
org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:37)
at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:558)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:206)
... 5 more
[2014-06-16 13:13:34,601][WARN ][discovery.zen.ping.multicast] [nw-es-201]
failed to receive confirmation on sent ping response to
[[nw-es-205][p9b19RfCT1uF7Ry2_eQXQw][nw-es-205.prod.pcln.com][inet[/172.21.88.116:9320]]{master=true}]
org.elasticsearch.transport.SendRequestTransportException:
[nw-es-205][inet[/172.21.88.116:9320]][discovery/zen/multicast]
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:219)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:190)
at
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.handleNodePingRequest(MulticastZenPing.java:462)
at
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.onMessage(MulticastZenPing.java:334)
at
org.elasticsearch.common.network.MulticastChannel$Plain$Receiver.run(MulticastChannel.java:363)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Maximum
number of bytes allocated exceeded: [836334432] (> 836318003)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:384)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:398)
at
org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:423)
at
org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:431)
at
org.elasticsearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:60)
at
org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:37)
at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:558)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:206)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3769ecc9-1119-4f3e-8131-76d4cfef24bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

spinscale · June 17, 2014, 6:08am

Hey,

do you have a full stack trace in one of your log files? Also, are you
creating lots of new connections all the time and then stop them abruptly
(just a wild guess here)?

--Alex

On Mon, Jun 16, 2014 at 11:18 PM, Sriram Kannan kesri8@gmail.com wrote:

I am getting this error constantly and the 5 nodes in the cluster goes
down one by one. Is there a way to fix the issue? Could it be a corrupt
data issue?

[2014-06-16 13:13:34,422][WARN ][discovery.zen.ping.multicast] [nw-es-201]
failed to receive confirmation on sent ping response to
[[nw-es-202][xOi0MSQBQ4OoQjpuIDUjtA][nw-es-202.prod.pcln.com
][inet[/172.21.88.113:9320]]{master=true}]
org.elasticsearch.transport.SendRequestTransportException:
[nw-es-202][inet[/172.21.88.113:9320]][discovery/zen/multicast]
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:219)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:190)
at
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.handleNodePingRequest(MulticastZenPing.java:462)
at
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.onMessage(MulticastZenPing.java:334)
at
org.elasticsearch.common.network.MulticastChannel$Plain$Receiver.run(MulticastChannel.java:363)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Maximum
number of bytes allocated exceeded: [836334432] (> 836318003)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:384)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:398)
at
org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:423)
at
org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:431)
at
org.elasticsearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:60)
at
org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:37)
at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:558)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:206)
... 5 more
[2014-06-16 13:13:34,601][WARN ][discovery.zen.ping.multicast] [nw-es-201]
failed to receive confirmation on sent ping response to
[[nw-es-205][p9b19RfCT1uF7Ry2_eQXQw][nw-es-205.prod.pcln.com
][inet[/172.21.88.116:9320]]{master=true}]
org.elasticsearch.transport.SendRequestTransportException:
[nw-es-205][inet[/172.21.88.116:9320]][discovery/zen/multicast]
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:219)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:190)
at
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.handleNodePingRequest(MulticastZenPing.java:462)
at
org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.onMessage(MulticastZenPing.java:334)
at
org.elasticsearch.common.network.MulticastChannel$Plain$Receiver.run(MulticastChannel.java:363)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Maximum
number of bytes allocated exceeded: [836334432] (> 836318003)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:384)
at org.elasticsearch.common.util.BigArrays.validate(BigArrays.java:398)
at
org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:423)
at
org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:431)
at
org.elasticsearch.common.io.stream.BytesStreamOutput.(BytesStreamOutput.java:60)
at
org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.(ReleasableBytesStreamOutput.java:37)
at
org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:558)
at
org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:206)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3769ecc9-1119-4f3e-8131-76d4cfef24bb%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3769ecc9-1119-4f3e-8131-76d4cfef24bb%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9Uv3EOij7HwHMJRBSLBzkMgAP2pQN6CHo_gzJF-zWfbQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.