Invalid internal transport message format-while initializing Search-Guard

[madhu2103 (madhu)]

We have installed elasticsearch 6.4 and working fine.But when we are are trying to install search-Guard with keystore certs on same node we facing below issue.

sudo chmod +x ./sgadmin.sh && sudo ./sgadmin.sh -h ...[IP address] -cd ../sgconfig -cn abc -ts /etc/elasticsearch/truststore.jks -tspass ***** tsalias truststore -ks /etc/elasticsearch/keystore.jks -kspass ***** -nhnv

[INFO ][o.e.h.n.Netty4HttpServerTransport] [node-1] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[INFO ][o.e.n.Node ] [node-1] started
[INFO ][c.f.s.SearchGuardPlugin ] 0 Search Guard modules loaded so far:
[INFO ][o.e.l.LicenseService ] [node-1] license [**********************] mode [basic] - valid
[INFO ][o.e.g.GatewayService ] [node-1] recovered [0] indices into cluster_state
[WARN ][o.e.t.n.Netty4Transport ] [node-1] exception caught on transport layer [NettyTcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.1:48096}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (16,3,3,0)
at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1327) ~[elasticsearch-6.4.0.jar:6.4.0]

elasticsearch.yml

cluster.name: ABC
node.name: node-0
node.master: false
node.data: false
node.ingest: false
search.remote.connect: false
discovery.zen.ping.unicast.hosts: ["localhost IP"]

searchguard.disabled: true
xpack.security.enabled: false
#searchguard.ssl.http.enabled: false

---------------------------------- Search Guard SSL ----------------------------

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_filepath: /etc/elasticsearch/keystore.jks
searchguard.ssl.transport.keystore_alias: elasticsearch
searchguard.ssl.transport.keystore_password: *********
searchguard.ssl.transport.truststore_filepath: /etc/elasticsearch/truststore.jks
searchguard.ssl.transport.truststore_alias: truststore
searchguard.ssl.transport.truststore_password: *********
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false

---------------------------------- HTTP/REST layer SSL ----------------------------

searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: /etc/elasticsearch/keystore.jks
searchguard.ssl.http.keystore_alias: elasticsearch
searchguard.ssl.http.keystore_password: *********
searchguard.ssl.http.truststore_filepath: /etc/elasticsearch/truststore.jks
searchguard.ssl.http.truststore_alias: truststore
searchguard.ssl.http.truststore_password: *********
searchguard.nodes_dn:

• "CN=*********, OU=*********, O=*********, L=*********, ST=*********, C=*********"
  searchguard.authcz.admin_dn:
• "CN=*********, OU=*********, O=*********, L=*********, ST=*********, C=*********"

[dadoonet (David Pilato)]

Welcome. We don't support searchguard here so you may better ask on their forums?

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

[pushpendra08]

thanks david.
i will ask same question to searchguard forum.
but i have another doubt.

is it possible to configure elastic search with searchguard on same node with ssl(using keystore.jks)?

[dadoonet (David Pilato)]

Sorry @pushpendra08 but are you working with @madhu2103?

@pushpendra08

is it possible to configure Elasticsearch with searchguard on same node with ssl(using keystore.jks)?

I don't know. We have elasticsearch SSL for free starting from 6.8 and 7.1 so unless you are using the official features, I'm afraid we can't help here.

[pushpendra08]

Yes.working on same.
Now we got another error related to elastic search.

---

after setting searchguard.disabled to false we are getting error

Journalctl logs

[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node-0] timed out while retrying [cluster:monitor/health] after failure (timeout [30s])
[DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node-0] no known master node, scheduling a retry
: [2019-06-30T13:39:11,693][WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again

Elastic search logs:-

[WARN ][o.e.n.Node ] [node-0] timed out while waiting for initial discovery state - timeout: 30s
[INFO ][c.f.s.h.SearchGuardHttpServerTransport] [node-0] publish_address { . . . :9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200}, { . . . :9200}
[INFO ][o.e.n.Node ] [node-0] started
[INFO ][c.f.s.SearchGuardPlugin ] 4 Search Guard modules loaded so far: [Module [type=AUDITLOG, implementing class=com.floragunn.searchguard.auditlog.impl.AuditLogImpl], Module [type=DLSFLS, implementing class=com.floragunn.searchguard.configuration.SearchGuardFlsDlsIndexSearcherWrapper], Module [type=MULTITENANCY, implementing class=com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl], Module [type=REST_MANAGEMENT_API, implementing class=com.floragunn.searchguard.dlic.rest.api.SearchGuardRestApiActions]]
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[WARN ][o.e.d.z.ZenDiscovery ] [node-0] not enough master nodes discovered during pinging (found , but needed [-1]), pinging again
[DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [node-0] timed out while retrying [indices:admin/exists] after failure (timeout [1m])
[ERROR][c.f.s.c.IndexBaseConfigurationRepository] Failure while checking MasterNotDiscoveredException[null] index searchguard
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:223) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:317) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:244) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:573) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.4.0.jar:6.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
[2019-06-30T13:28:29,474][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [node-0] no known master node, scheduling a retry

Please sugest how to resolve this

[dadoonet (David Pilato)]

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

Here I can still see some traces coming from searchguard plugin. I'd recommend removing the plugin, try again, share full logs and configuration files (formatted).

[nkoothrappahli (Nanda Koothrappahli)]

Seems the issue is with not having any master eligible or data node

node.master: false
node.data: false