Weird log entries in elasticsearch.log

Elastic Stack Elasticsearch
1

I got some weird errors in my ES log...

[2012-12-27 15:37:25,267][WARN ][monitor.jvm ] [Locus]
[gc][ParNew][2676426][42057] duration [9.4s], collections [1]/[3.8m], total
[9.4s]/[40.1m], memory [933.9mb]->[740.5mb]/[1.9gb], all_pools {[Code
Cache] [12.9mb]->[12.9mb]/[48mb]}{[Par Eden Space]
[231.5mb]->[152kb]/[266.2mb]}{[Par Survivor Space]
[33.2mb]->[33.2mb]/[33.2mb]}{[CMS Old Gen]
[669.1mb]->[707.1mb]/[1.6gb]}{[CMS Perm Gen] [33.2mb]->[33.2mb]/[82mb]}

Does anyone know what that means?

Then I got a lot of:

[2012-12-27 17:21:01,243][WARN ][http.netty ] [Locus] Caught
excep
tion while handling client http traffic, closing connection [id:
0x6d71402b, /10
.177.228.62:60136 :> /10.180.158.120:9200]
java.lang.IllegalStateException: received HttpChunk without HttpMessage
at
org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator
.messageReceived(HttpChunkAggregator.java:116)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.h
andleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUps
tream(DefaultChannelPipeline.java:563)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$Default
ChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(C
hannels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfol
dAndFireMessageReceived(FrameDecoder.java:420)

After all of the weird HttpChunk things ES stopped working.

Thanks in advance for any assistance!

  • Allison

--

2

The first part is a warning message that Java garbage collection took
longer than expected. Are you running queries with faceting, sorting
or scripting? You see these warnings often? Perhaps the es process
needs more memory. You can check this with the node stats api:

(use the jvm flag)

I think the last error could be a result of the first warning. Usually
when memory becomes scarce ES nodes can become not responsive.

Martijn

On 28 December 2012 17:12, Allison A.
allisonandrews@feranassociates.com wrote:

I got some weird errors in my ES log...

[2012-12-27 15:37:25,267][WARN ][monitor.jvm ] [Locus]
[gc][ParNew][2676426][42057] duration [9.4s], collections [1]/[3.8m], total
[9.4s]/[40.1m], memory [933.9mb]->[740.5mb]/[1.9gb], all_pools {[Code Cache]
[12.9mb]->[12.9mb]/[48mb]}{[Par Eden Space]
[231.5mb]->[152kb]/[266.2mb]}{[Par Survivor Space]
[33.2mb]->[33.2mb]/[33.2mb]}{[CMS Old Gen]
[669.1mb]->[707.1mb]/[1.6gb]}{[CMS Perm Gen] [33.2mb]->[33.2mb]/[82mb]}

Does anyone know what that means?

Then I got a lot of:

[2012-12-27 17:21:01,243][WARN ][http.netty ] [Locus] Caught
excep
tion while handling client http traffic, closing connection [id: 0x6d71402b,
/10
.177.228.62:60136 :> /10.180.158.120:9200]
java.lang.IllegalStateException: received HttpChunk without HttpMessage
at
org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator
.messageReceived(HttpChunkAggregator.java:116)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.h
andleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUps
tream(DefaultChannelPipeline.java:563)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$Default
ChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(C
hannels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfol
dAndFireMessageReceived(FrameDecoder.java:420)

After all of the weird HttpChunk things ES stopped working.

Thanks in advance for any assistance!

  • Allison

--

--
Met vriendelijke groet,

Martijn van Groningen

--