Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (0,0,0,28);

kingjianle · August 1, 2022, 2:01am

Elasticsearch version (6.8.0):

Plugins installed: [elasticsearch-analysis-ik]6.8.0

JVM version (java version "1.8.0_333"):

OS version (uname -a if on a Unix-like system):Linux localhost.localdomain 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux By docker

Springboot version (2.1.4.RELEASE)

Description of the problem including expected versus actual behavior:
This happens all the time: Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (0,0,0,28);
Why it happend? How to stop it?

somethings specially:
I copided the folder (elasticsearch/data/nodes) from Elasticsearch6.4.3 to the samepath of Elasticsearch6.8.0,Because,I want to keep the datas before;
This exactly is a better method than others,Because it is so simple,and I keeped my datas;
But even I removed the folder,the problem is still exist.

Why it happend? How to stop it? Does it slow down the Linux OS or cause other bad things?

Provide logs (if relevant):

[2022-07-30T07:46:32,019][WARN ][o.e.t.TcpTransport       ] [27nrFZT] exception caught on transport layer [Netty4TcpChannel{localAddress=/172.17.0.7:9300, remoteAddress=/172.17.0.1:52634}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (0,0,0,28)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (0,0,0,28)
	at org.elasticsearch.transport.TcpTransport.readHeaderBuffer(TcpTransport.java:851) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.TcpTransport.readMessageLength(TcpTransport.java:837) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:40) ~[transport-netty4-client-6.8.0.jar:6.8.0]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	... 20 more
[2022-07-30T07:46:27,014][WARN ][o.e.t.TcpTransport       ] [27nrFZT] exception caught on transport layer [Netty4TcpChannel{localAddress=/172.17.0.7:9300, remoteAddress=/172.17.0.1:52622}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (0,0,0,28)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (0,0,0,28)
	at org.elasticsearch.transport.TcpTransport.readHeaderBuffer(TcpTransport.java:851) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.TcpTransport.readMessageLength(TcpTransport.java:837) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:40) ~[transport-netty4-client-6.8.0.jar:6.8.0]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	... 19 more

warkolm · August 1, 2022, 6:19am

Welcome to our community!

What sort of applications talk to your cluster?

kingjianle · August 1, 2022, 7:35am

java application;
not a cluster,a single node;
spring-boot-starter-data-elasticsearch: 2.1.4.RELEASE

kingjianle · August 1, 2022, 10:49am

I guess it maybe caused by ssl,But can not confirm.
because,the request became from nginx that Configured with ssl.

I'm not sure because the error codes are different: got (0,0,0,28);

warkolm · August 2, 2022, 1:28am

A single node is still technically a cluster, just of one node.

Does it happen if you remove nginx?

kingjianle · August 2, 2022, 2:03am

I can not do that, because this is a online project.
The nginx is being accessed,all the time.

warkolm · August 2, 2022, 2:04am

Can you bypass nginx and talk directly to Elasticsearch?

kingjianle · August 2, 2022, 2:16am

Yes, I can .
When I accessed the site:http://192.168.88.21:9200, I got the correct response.

name	"27nrFZT"
cluster_name	"docker-cluster"
cluster_uuid	"mAGnpJapQXSdIAzgYzS2NQ"
version	
number	"6.8.0"
build_flavor	"default"
build_type	"docker"
build_hash	"65b6179"
build_date	"2019-05-15T20:06:13.172855Z"
build_snapshot	false
lucene_version	"7.7.0"
minimum_wire_compatibility_version	"5.6.0"
minimum_index_compatibility_version	"5.0.0"
tagline	"You Know, for Search"

When I accessed the site:https://192.168.88.21:9200, I got " * Failed to set up a secure connection. Procedure";

warkolm · August 2, 2022, 2:51am

Elasticsearch 6.8 is EOL and you need to upgrade ASAP.

If it's working when talking to Elasticsearch directly, then that would suggest that there's something wrong with your proxy, which is outside the scope of what we can help with.

kingjianle · August 2, 2022, 3:10am

I am not sure it is or not working when talking to Elasticsearch directly,because the warning is printing all the time , and I cann't stop it.
And the elasticsearch is functioning normally,when I search or add something.
But, there are warnings all the time , like above.
I want to know: Why it happend? How to stop it? Does it slow down the Linux OS or cause other bad things?
I will check my proxy,and thank you , sir.

warkolm · August 2, 2022, 3:12am

What is the 172.17.0.1 host?

kingjianle · August 2, 2022, 3:22am

It is docker's IPAM gateway.

name  scope  drive  Can be attached   IPAM drive    IPAM subnet      IPAM gateway    ownership
bridge  local   bridge    false                    default         172.17.0.0/16      172.17.0.1           public

system · August 30, 2022, 3:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.