java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)

Elastic Stack Elasticsearch
1

hi , i am updating my xpack security in all my cluster nodes , by using this yml.file,

action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
    xpack.security.audit.enabled: true
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate

    xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12 
    xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
    action.destructive_requires_name: true

    xpack:
      security:
        authc:
          realms:
            native1:
              type: native
              order: 0
    after that i tried to restart one by one node , it is working fine for three nodes , but when i am restarting 4 th node i am getting below error .in all nodes .



    `[WARN ][o.e.t.TcpTransport       ] [node] exception caught on transport layer [Netty4TcpChannel{localAddress=/x.x.x.x:9300, remoteAddress=/x.x.x.x:54938}], closing connection`

    io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)
    	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
    	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
    	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]

> Blockquote

4 spaces

2

after that i tried to restart one by one node , it is working fine for three nodes , but when i am restarting 4 th node i am getting above error .in all nodes .

3

This means the node received encrypted (SSL) traffic but does not have SSL enabled.

4

but i have enabled ssl in yml configuration
xpack.security.transport.ssl.enabled: true

5

is there any other places i need to change?

6

failed to connect to node. (tried [115] times)
org.elasticsearch.transport.ConnectTransportException: [node1][x.x.x.x:9300] general node connection failure , i am also seeing this message in logs

7

Have you created and configured certificates? I do not see this n you config snippet above.

8

The only explanation I can think of is that you haven't actually done this on all your nodes.

9

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate

xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
yes ,i have first generated ca-certificate in one servers , after that i copies ca certificates to all the nodes and then i started generated node certificate for all nodes .

10

yes, i did above changes in all the nodes , but when i am restarting one by one server it is works fine for three nodes (but total i have 7 nodes), restarting 4 th node i am getting above error .

11

If you haven't restarted all the nodes after adjusting their settings then you haven't really enabled SSL on them. You have to restart all the nodes to pick up the new settings.

12

hi sir , i have restarted all the nodes , but i am getting some error
"secure setting cannot be used with tribe client node xpack.security.transport.ssl.keystore,secure_passwd "

this is the yml file configuration for estribe node
tribe:
t1:
cluster.name: Data1
discovery.zen.ping.unicast.hosts: ["XX.x.x.x:9300","x.x.x.x:9300"]
xpack.security.transport.ssl.verification_mode: certificate
#xpack.security.transport.ssl.keystore.path: certs/tribe.p12
#xpack.security.transport.ssl.truststore.path: certs/tribe.p12
xpack.security.transport.ssl.enabled: true
t2:
cluster.name: data2
discovery.zen.ping.unicast.hosts: ["x.x.x.x:9300","x.x.x.:9300"]
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/tribe.p12
xpack.security.transport.ssl.truststore.path: certs/tribe.p12
xpack.security.transport.ssl.enabled: true
tribe.on_conflict: data2

java.lang.IllegalArgumentException: Secure settings [xpack.security.transport.ssl.keystore.secure_password, xpack.security.transport.ssl.truststore.secure_password] cannot be used with tribe client node

is there any wrong with my configuration , please correct me .
certificate is in PKCS#12 format

i used this link to setup https://www.elastic.co/guide/en/elasticsearch/reference/6.8/tribe-node-configuring.html , but in the link there are using PEM Format , accoring to that i changed to PKCS format

13

Which version of Elasticsearch are you using? What is the topology you are trying to deploy (I see something about a tribe node)?

14

elastic search - 6.8.13 , two different clusters will need to access tribe node , can you please share me the tribe node yml configuration .for PKCS#12 format

15

Tribe nodes are being deprecated, so I would recommend switching to cross-cluster search instead. I have not used tribe nodes in a very long time so can unfortunately not help.

16

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 45530000002c000000000009379f08004d3603000016696e7465726e616c3a7463702f68616e647368616b650004f796f302
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1182) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
... 15 more
indent preformatted text by 4 spaces

17

i used
this time this configuration PEM format
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /home/es/config/node01.key
xpack.security.transport.ssl.certificate: /home/es/config/node01.crt
xpack.security.transport.ssl.certificate_authorities: [ "/home/es/config/ca.crt" ]

tribe node is all so connected , i set xpack.security.transport.ssl.enabled: true in all the nodes , i restarted all the nodes , but still i am getting error Preformatted text
[2020-12-23T15:38:45,409][WARN ][o.e.t.TcpTransport ] [node] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/x.x.x.:38468}], closing connection
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 45530000002c000000000009379f08004d3603000016696e7465726e616c3a7463702f68616e647368616b650004f796f302
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.