# java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)
**URL:** https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679
**Category:** Elasticsearch
**Created:** [December 4, 2020, 4:50pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679 "2020-12-04T16:50:58Z")
**Posts on this page:** 18
**Page:** 1
### Author:  [@prashanth\_sri](https://discuss.elastic.co/u/prashanth_sri)
#### Post date: [December 4, 2020, 4:50pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/1 "2020-12-04T16:50:59Z")
> hi , i am updating my xpack security in all my cluster nodes , by using this yml.file,
```
action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
xpack.security.audit.enabled: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
action.destructive_requires_name: true
xpack:
security:
authc:
realms:
native1:
type: native
order: 0
after that i tried to restart one by one node , it is working fine for three nodes , but when i am restarting 4 th node i am getting below error .in all nodes .
`[WARN][o.e.t.TcpTransport] [node] exception caught on transport layer [Netty4TcpChannel{localAddress=/x.x.x.x:9300, remoteAddress=/x.x.x.x:54938}], closing connection`
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
```
`> Blockquote`
4 spaces
---
### Author:  [@prashanth\_sri](https://discuss.elastic.co/u/prashanth_sri)
#### Post date: [December 4, 2020, 5:03pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/2 "2020-12-04T17:03:35Z")
> [@prashanth\_sri](#):
>
> _ **`after that i tried to restart one by one node , it is working fine for three nodes , but when i am restarting 4 th node i am getting below error .in all nodes .`** _
after that i tried to restart one by one node , it is working fine for three nodes , but when i am restarting 4 th node i am getting above error .in all nodes .
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 4, 2020, 5:37pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/3 "2020-12-04T17:37:24Z")
> [@prashanth\_sri](#):
>
> ```auto
> invalid internal transport message format, got (16,3,3,0)
>
> ```
This means the node received encrypted (SSL) traffic but does not have SSL enabled.
---
### Author:  [@prashanth\_sri](https://discuss.elastic.co/u/prashanth_sri)
#### Post date: [December 6, 2020, 4:56am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/4 "2020-12-06T04:56:57Z")
but i have enabled ssl in yml configuration
xpack.security.transport.ssl.enabled: true
---
### Author:  [@prashanth\_sri](https://discuss.elastic.co/u/prashanth_sri)
#### Post date: [December 6, 2020, 4:57am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/5 "2020-12-06T04:57:42Z")
is there any other places i need to change?
---
### Author:  [@prashanth\_sri](https://discuss.elastic.co/u/prashanth_sri)
#### Post date: [December 6, 2020, 5:06am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/6 "2020-12-06T05:06:32Z")
failed to connect to node. (tried [115] times)
org.elasticsearch.transport.ConnectTransportException: [node1][x.x.x.x:9300] general node connection failure , i am also seeing this message in logs
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 6, 2020, 6:54am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/7 "2020-12-06T06:54:39Z")
Have you created and configured certificates? I do not see this n you config snippet above.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 6, 2020, 12:24pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/8 "2020-12-06T12:24:36Z")
> [@prashanth\_sri](#):
>
> but i have enabled ssl in yml configuration
> xpack.security.transport.ssl.enabled: true
The only explanation I can think of is that you haven't actually done this on all your nodes.
---
### Author:  [@prashanth\_sri](https://discuss.elastic.co/u/prashanth_sri)
#### Post date: [December 6, 2020, 1:02pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/9 "2020-12-06T13:02:04Z")
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification\_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
yes ,i have first generated ca-certificate in one servers , after that i copies ca certificates to all the nodes and then i started generated node certificate for all nodes .
---
### Author:  [@prashanth\_sri](https://discuss.elastic.co/u/prashanth_sri)
#### Post date: [December 6, 2020, 3:58pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/10 "2020-12-06T15:58:07Z")
yes, i did above changes in all the nodes , but when i am restarting one by one server it is works fine for three nodes (but total i have 7 nodes), restarting 4 th node i am getting above error .
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 6, 2020, 5:17pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/11 "2020-12-06T17:17:03Z")
If you haven't restarted all the nodes after adjusting their settings then you haven't really enabled SSL on them. You have to restart all the nodes to pick up the new settings.
---
### Author:  [@prashanth\_sri](https://discuss.elastic.co/u/prashanth_sri)
#### Post date: [December 18, 2020, 4:21pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/12 "2020-12-18T16:21:49Z")
hi sir , i have restarted all the nodes , but i am getting some error
"secure setting cannot be used with tribe client node xpack.security.transport.ssl.keystore,secure\_passwd "
this is the yml file configuration for estribe node
tribe:
t1:
cluster.name: Data1
discovery.zen.ping.unicast.hosts: ["XX.x.x.x:9300","x.x.x.x:9300"]
xpack.security.transport.ssl.verification\_mode: certificate
#xpack.security.transport.ssl.keystore.path: certs/tribe.p12
#xpack.security.transport.ssl.truststore.path: certs/tribe.p12
xpack.security.transport.ssl.enabled: true
t2:
cluster.name: data2
discovery.zen.ping.unicast.hosts: ["x.x.x.x:9300","x.x.x.:9300"]
xpack.security.transport.ssl.verification\_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/tribe.p12
xpack.security.transport.ssl.truststore.path: certs/tribe.p12
xpack.security.transport.ssl.enabled: true
tribe.on\_conflict: data2
java.lang.IllegalArgumentException: Secure settings [xpack.security.transport.ssl.keystore.secure\_password, xpack.security.transport.ssl.truststore.secure\_password] cannot be used with tribe client node
is there any wrong with my configuration , please correct me .
certificate is in PKCS#12 format
i used this link to setup [https://www.elastic.co/guide/en/elasticsearch/reference/6.8/tribe-node-configuring.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/tribe-node-configuring.html) , but in the link there are using PEM Format , accoring to that i changed to PKCS format
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 18, 2020, 5:29pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/13 "2020-12-18T17:29:05Z")
Which version of Elasticsearch are you using? What is the topology you are trying to deploy (I see something about a tribe node)?
---
### Author:  [@prashanth\_sri](https://discuss.elastic.co/u/prashanth_sri)
#### Post date: [December 18, 2020, 5:33pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/14 "2020-12-18T17:33:29Z")
elastic search - 6.8.13 , two different clusters will need to access tribe node , can you please share me the tribe node yml configuration .for PKCS#12 format
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 18, 2020, 5:40pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/15 "2020-12-18T17:40:37Z")
Tribe nodes are being deprecated, so I would recommend switching to [cross-cluster search](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/cross-cluster-configuring.html) instead. I have not used tribe nodes in a very long time so can unfortunately not help.
---
### Author:  [@prashanth\_sri](https://discuss.elastic.co/u/prashanth_sri)
#### Post date: [December 24, 2020, 3:50am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/16 "2020-12-24T03:50:57Z")
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 45530000002c000000000009379f08004d3603000016696e7465726e616c3a7463702f68616e647368616b650004f796f302
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1182) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
... 15 more
indent preformatted text by 4 spaces
---
### Author:  [@prashanth\_sri](https://discuss.elastic.co/u/prashanth_sri)
#### Post date: [December 24, 2020, 3:55am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/17 "2020-12-24T03:55:13Z")
i used
this time this configuration PEM format
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification\_mode: certificate
xpack.security.transport.ssl.key: /home/es/config/node01.key
xpack.security.transport.ssl.certificate: /home/es/config/node01.crt
xpack.security.transport.ssl.certificate\_authorities: ["/home/es/config/ca.crt"]
tribe node is all so connected , i set xpack.security.transport.ssl.enabled: true in all the nodes , i restarted all the nodes , but still i am getting error Preformatted text
[2020-12-23T15:38:45,409][WARN][o.e.t.TcpTransport] [node] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/x.x.x.:38468}], closing connection
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 45530000002c000000000009379f08004d3603000016696e7465726e616c3a7463702f68616e647368616b650004f796f302
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_242]
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 21, 2021, 3:55am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format-got-16-3-3-0/257679/18 "2021-01-21T03:55:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.