[WARN] http.netty "Invalid version format "

There are so many warning message. about 7k message per a day.

two message seems a set of error.

[2015-11-16 10:36:30,639][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x96e95426, /10.98.168.209:36733 => /10.98.168.207:9200]
java.lang.IllegalArgumentException: invalid version format: /STATS HTTP/1.1
[2015-11-16 10:36:30,639][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x96e95426, /10.98.168.209:36733 :> /10.98.168.207:9200]
java.lang.IllegalArgumentException: invalid version format: (X86_64-REDHAT-LINUX-GNU) LIBCURL/7.19.7 NSS/3.13.1.0 ZLIB/1.2.3 LIBIDN/1.18 LIBSSH2/1.2.2

full error message

[2015-11-16 10:36:30,639][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x96e95426, /10.98.168.209:36733 => /10.98.168.207:9200]
java.lang.IllegalArgumentException: invalid version format: /STATS HTTP/1.1
at org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:94)
at org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:62)
at org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75)
at org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:191)
at org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:102)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at java.lang.Thread.run(Thread.java:745)
[2015-11-16 10:36:30,639][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x96e95426, /10.98.168.209:36733 :> /10.98.168.207:9200]
java.lang.IllegalArgumentException: invalid version format: (X86_64-REDHAT-LINUX-GNU) LIBCURL/7.19.7 NSS/3.13.1.0 ZLIB/1.2.3 LIBIDN/1.18 LIBSSH2/1.2.2
at org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:94)
at org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:62)
at org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75)
at org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:191)
at org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:102)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:554)
at

  • all of nodes running with es 1.7.2
  • java 1.8.0.45
  • OS : centos release 6.3 final

what about this message warning? and how can I fix?

Looks like something from 10.98.168.209 is connecting and passing in bad data.
You'd have to check that host to see what.

Looks like something from 10.98.168.209 is connecting and passing in bad data.
You'd have to check that host to see what.

not only between 10.98.168.209:36733 => /10.98.168.207:9200
but also between every node make warning message.

$ cat DodolLauncherES.log | grep "closing connection"
[2015-11-16 00:00:30,067][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0xfb2bc0f0, /10.98.168.208:42966 => /10.98.168.207:9200]
[2015-11-16 00:00:30,069][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0xfb2bc0f0, /10.98.168.208:42966 :> /10.98.168.207:9200]
[2015-11-16 00:00:30,181][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x0b8a1136, /10.98.168.210:35181 => /10.98.168.207:9200]
[2015-11-16 00:00:30,182][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x0b8a1136, /10.98.168.210:35181 :> /10.98.168.207:9200]
[2015-11-16 00:00:30,408][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x2f804969, /10.98.168.207:57296 => /10.98.168.207:9200]
[2015-11-16 00:00:30,409][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x2f804969, /10.98.168.207:57296 :> /10.98.168.207:9200]
[2015-11-16 00:00:30,564][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0xeb5acb97, /10.98.168.209:34009 => /10.98.168.207:9200]
[2015-11-16 00:00:30,565][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0xeb5acb97, /10.98.168.209:34009 :> /10.98.168.207:9200]
[2015-11-16 00:00:30,596][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x9dd4227d, /10.98.168.211:56323 => /10.98.168.207:9200]
[2015-11-16 00:00:30,596][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x9dd4227d, /10.98.168.211:56323 :> /10.98.168.207:9200]
[2015-11-16 00:01:30,061][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x7b54e6d1, /10.98.168.208:42970 => /10.98.168.207:9200]
[2015-11-16 00:01:30,062][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x7b54e6d1, /10.98.168.208:42970 :> /10.98.168.207:9200]
[2015-11-16 00:01:30,181][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x26983790, /10.98.168.210:35185 => /10.98.168.207:9200]

Ok, so what else is on those hosts?
Are you sending non-HTTP data to the HTTP port?