[WARN] http.netty "Invalid version format "

Elastic Stack Elasticsearch
1

There are so many warning message. about 7k message per a day.

two message seems a set of error.

[2015-11-16 10:36:30,639][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x96e95426, /10.98.168.209:36733 => /10.98.168.207:9200]
java.lang.IllegalArgumentException: invalid version format: /STATS HTTP/1.1
[2015-11-16 10:36:30,639][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x96e95426, /10.98.168.209:36733 :> /10.98.168.207:9200]
java.lang.IllegalArgumentException: invalid version format: (X86_64-REDHAT-LINUX-GNU) LIBCURL/7.19.7 NSS/3.13.1.0 ZLIB/1.2.3 LIBIDN/1.18 LIBSSH2/1.2.2

full error message

[2015-11-16 10:36:30,639][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x96e95426, /10.98.168.209:36733 => /10.98.168.207:9200]
java.lang.IllegalArgumentException: invalid version format: /STATS HTTP/1.1
at org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:94)
at org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:62)
at org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75)
at org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:191)
at org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:102)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at java.lang.Thread.run(Thread.java:745)
[2015-11-16 10:36:30,639][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x96e95426, /10.98.168.209:36733 :> /10.98.168.207:9200]
java.lang.IllegalArgumentException: invalid version format: (X86_64-REDHAT-LINUX-GNU) LIBCURL/7.19.7 NSS/3.13.1.0 ZLIB/1.2.3 LIBIDN/1.18 LIBSSH2/1.2.2
at org.elasticsearch.common.netty.handler.codec.http.HttpVersion.(HttpVersion.java:94)
at org.elasticsearch.common.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:62)
at org.elasticsearch.common.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75)
at org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:191)
at org.elasticsearch.common.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:102)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
at org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.cleanup(ReplayingDecoder.java:554)
at

  • all of nodes running with es 1.7.2
  • java 1.8.0.45
  • OS : centos release 6.3 final

what about this message warning? and how can I fix?

2

Looks like something from 10.98.168.209 is connecting and passing in bad data.
You'd have to check that host to see what.

3 
Looks like something from 10.98.168.209 is connecting and passing in bad data.
You'd have to check that host to see what.

not only between 10.98.168.209:36733 => /10.98.168.207:9200
but also between every node make warning message.

$ cat DodolLauncherES.log | grep "closing connection"
[2015-11-16 00:00:30,067][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0xfb2bc0f0, /10.98.168.208:42966 => /10.98.168.207:9200]
[2015-11-16 00:00:30,069][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0xfb2bc0f0, /10.98.168.208:42966 :> /10.98.168.207:9200]
[2015-11-16 00:00:30,181][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x0b8a1136, /10.98.168.210:35181 => /10.98.168.207:9200]
[2015-11-16 00:00:30,182][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x0b8a1136, /10.98.168.210:35181 :> /10.98.168.207:9200]
[2015-11-16 00:00:30,408][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x2f804969, /10.98.168.207:57296 => /10.98.168.207:9200]
[2015-11-16 00:00:30,409][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x2f804969, /10.98.168.207:57296 :> /10.98.168.207:9200]
[2015-11-16 00:00:30,564][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0xeb5acb97, /10.98.168.209:34009 => /10.98.168.207:9200]
[2015-11-16 00:00:30,565][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0xeb5acb97, /10.98.168.209:34009 :> /10.98.168.207:9200]
[2015-11-16 00:00:30,596][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x9dd4227d, /10.98.168.211:56323 => /10.98.168.207:9200]
[2015-11-16 00:00:30,596][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x9dd4227d, /10.98.168.211:56323 :> /10.98.168.207:9200]
[2015-11-16 00:01:30,061][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x7b54e6d1, /10.98.168.208:42970 => /10.98.168.207:9200]
[2015-11-16 00:01:30,062][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x7b54e6d1, /10.98.168.208:42970 :> /10.98.168.207:9200]
[2015-11-16 00:01:30,181][WARN ][http.netty ] [xanalytics501] Caught exception while handling client http traffic, closing connection [id: 0x26983790, /10.98.168.210:35185 => /10.98.168.207:9200]

4

Ok, so what else is on those hosts?
Are you sending non-HTTP data to the HTTP port?