Invalid internal transport message format-while initializing Search-Guard

madhu2103 · June 30, 2019, 6:12am

We have installed elasticsearch 6.4 and working fine.But when we are are trying to install search-Guard with keystore certs on same node we facing below issue.

sudo chmod +x ./sgadmin.sh && sudo ./sgadmin.sh -h ...[IP address] -cd ../sgconfig -cn abc -ts /etc/elasticsearch/truststore.jks -tspass ***** tsalias truststore -ks /etc/elasticsearch/keystore.jks -kspass ***** -nhnv

[INFO ][o.e.h.n.Netty4HttpServerTransport] [node-1] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[INFO ][o.e.n.Node ] [node-1] started
[INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:
[INFO ][o.e.l.LicenseService ] [node-1] license [**********************] mode [basic] - valid
[INFO ][o.e.g.GatewayService ] [node-1] recovered [0] indices into cluster_state
[WARN ][o.e.t.n.Netty4Transport ] [node-1] exception caught on transport layer [NettyTcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:48096}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)
at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1327) ~[elasticsearch-6.4.0.jar:6.4.0]

elasticsearch.yml

cluster.name: ABC
node.name: node-0
node.master: false
node.data: false
node.ingest: false
search.remote.connect: false
discovery.zen.ping.unicast.hosts: ["localhost IP"]

searchguard.disabled: true
xpack.security.enabled: false
#searchguard.ssl.http.enabled: false

---------------------------------- Search Guard SSL ----------------------------

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_filepath: /etc/elasticsearch/keystore.jks
searchguard.ssl.transport.keystore_alias: elasticsearch
searchguard.ssl.transport.keystore_password: *********
searchguard.ssl.transport.truststore_filepath: /etc/elasticsearch/truststore.jks
searchguard.ssl.transport.truststore_alias: truststore
searchguard.ssl.transport.truststore_password: *********
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false

---------------------------------- HTTP/REST layer SSL ----------------------------

searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: /etc/elasticsearch/keystore.jks
searchguard.ssl.http.keystore_alias: elasticsearch
searchguard.ssl.http.keystore_password: *********
searchguard.ssl.http.truststore_filepath: /etc/elasticsearch/truststore.jks
searchguard.ssl.http.truststore_alias: truststore
searchguard.ssl.http.truststore_password: *********
searchguard.nodes_dn:

"CN=*********, OU=*********, O=*********, L=*********, ST=*********, C=*********"
searchguard.authcz.admin_dn:
"CN=*********, OU=*********, O=*********, L=*********, ST=*********, C=*********"

dadoonet · June 30, 2019, 9:26am

Welcome. We don't support searchguard here so you may better ask on their forums?

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

pushpendra08 · June 30, 2019, 10:32am

thanks david.
i will ask same question to searchguard forum.
but i have another doubt.

is it possible to configure elastic search with searchguard on same node with ssl(using keystore.jks)?

dadoonet · June 30, 2019, 11:21am

Sorry @pushpendra08 but are you working with @madhu2103?

I don't know. We have elasticsearch SSL for free starting from 6.8 and 7.1 so unless you are using the official features, I'm afraid we can't help here.

pushpendra08 · June 30, 2019, 3:00pm

Yes.working on same.
Now we got another error related to elastic search.

after setting searchguard.disabled to false we are getting error

Journalctl logs

[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node-0] timed out while retrying [cluster:monitor/health] after failure (timeout [30s])
[DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node-0] no known master node, scheduling a retry
: [2019-06-30T13:39:11,693][WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again

Elastic search logs:-

[WARN ][o.e.n.Node ] [node-0] timed out while waiting for initial discovery state - timeout: 30s
[INFO ][c.f.s.h.SearchGuardHttpServerTransport] [node-0] publish_address { . . . :9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}, { . . . :9200}
[INFO ][o.e.n.Node ] [node-0] started
[INFO ][c.f.s.SearchGuardPlugin ] 4 Search Guard modules loaded so far: [Module [type=AUDITLOG, implementing class=com.floragunn.searchguard.auditlog.impl.AuditLogImpl], Module [type=DLSFLS, implementing class=com.floragunn.searchguard.configuration.SearchGuardFlsDlsIndexSearcherWrapper], Module [type=MULTITENANCY, implementing class=com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl], Module [type=REST_MANAGEMENT_API, implementing class=com.floragunn.searchguard.dlic.rest.api.SearchGuardRestApiActions]]
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [node-0] timed out while retrying [indices:admin/exists] after failure (timeout [1m])
[ERROR][c.f.s.c.IndexBaseConfigurationRepository] Failure while checking MasterNotDiscoveredException[null] index searchguard
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:223) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:317) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:244) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:573) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.4.0.jar:6.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2019-06-30T13:28:29,474][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node-0] no known master node, scheduling a retry

Please sugest how to resolve this

dadoonet · June 30, 2019, 4:39pm

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

Here I can still see some traces coming from searchguard plugin. I'd recommend removing the plugin, try again, share full logs and configuration files (formatted).

nkoothrappahli · June 30, 2019, 9:17pm

Seems the issue is with not having any master eligible or data node

node.master: false
node.data: false

system · July 28, 2019, 9:17pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.