java.io.StreamCorruptedException

Elastic Stack Elasticsearch
1

Hi ,

When I am trying to connect DataNode with Master Node Getting error below as written below:

{"type": "server", "timestamp": "2019-08-30T12:08:21,959+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "prafull_singh",  "message": "master not discovered yet: have discovered [{prafull_singh}{shKCd54aQFef7Vt6kMNkLQ}{J6YhGHQwTsCfnuF_c2WWyg}{172.18.0.2}{172.18.0.2:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [34.244.248.50:9200] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0"  }

prafull | {"type": "server", "timestamp": "2019-08-30T12:08:29,972+0000", "level": "WARN", "component": "o.e.t.TcpTransport", "cluster.name": "docker-cluster", "node.name": "prafull_singh", "message": "exception caught on transport layer [Netty4TcpChannel{localAddress=/172.18.0.2:41170, remoteAddress=/34.XXX.XXX.XXX:9200}], closing connection" ,
prafull | "stacktrace": ["io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)",
prafull | "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at java.lang.Thread.run(Thread.java:835) [?:?]",
prafull | "Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)",
prafull | "at org.elasticsearch.transport.TcpTransport.readHeaderBuffer(TcpTransport.java:745) ~[elasticsearch-7.3.1.jar:7.3.1]",
prafull | "at org.elasticsearch.transport.TcpTransport.readMessageLength(TcpTransport.java:731) ~[elasticsearch-7.3.1.jar:7.3.1]",
prafull | "at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:40) ~[transport-netty4-client-7.3.1.jar:7.3.1]",
prafull | "at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]",
prafull | "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]",
prafull | "... 20 more"] }
prafull | {"type": "server", "timestamp": "2019-08-30T12:08:29,981+0000", "level": "WARN", "component": "o.e.t.TcpTransport", "cluster.name": "docker-cluster", "node.name": "prafull_singh", "message": "exception caught on transport layer [Netty4TcpChannel{localAddress=/172.18.0.2:41170, remoteAddress=/34.XXX.XXX.XXX:9200}], closing connection" ,
prafull | "stacktrace": ["io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)",
prafull | "at

dataNode docker-compose.yml

version: '2.2'
services:
prafull:
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.1
container_name: prafull
environment:
- node.name=prafull_singh
- xpack.monitoring.enabled=false
- node.master=false
- node.data=true
- node.ingest=false
- cluster.name=docker-cluster
- discovery.seed_hosts=34.XX.XX.XX:9200
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms254m -Xmx254m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- prafull:/usr/share/elasticsearch/data
networks:
- esnet

volumes:
prafull:
driver: local

networks:
esnet:

2

Elastic communicated internally on port 9300 and you have configured port 9200 for discovery.seed_hosts. I also suspect you will need to increase the size of the heap to at least the double in order to be able to do anything useful.

3

Hi Christian,

After updating both port and heap size, data node is up but not able to join master nodes cluster. When I am running
curl -X GET "IP:9200/_cat/master" it gave me proper master node detail. Find Error Detai Below

{"type": "server", "timestamp": "2019-09-02T12:59:05,824+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2019-09-02T12:59:15,825+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2019-09-02T12:59:25,825+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2019-09-02T12:59:35,826+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2019-09-02T12:59:45,827+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2019-09-02T12:59:55,828+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2019-09-02T13:00:05,829+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2019-09-02T13:00:15,830+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
{"type": "server", "timestamp": "2019-09-02T13:00:19,217+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "es06", "message": "stopping ..." }
{"type": "server", "timestamp": "2019-09-02T13:00:19,238+0000", "level": "INFO", "component": "o.e.x.w.WatcherService", "cluster.name": "docker-cluster", "node.name": "es06", "message": "stopping watch service, reason [shutdown initiated]" }
{"type": "server", "timestamp": "2019-09-02T13:00:19,348+0000", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "es06", "message": "[controller/102] [Main.cc@150] Ml controller exiting" }
{"type": "server", "timestamp": "2019-09-02T13:00:19,349+0000", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "docker-cluster", "node.name": "es06", "message": "Native controller process has stopped - no new native processes can be started" }
{"type": "server", "timestamp": "2019-09-02T13:00:23,025+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "es06", "message": "stopped" }
{"type": "server", "timestamp": "2019-09-02T13:00:23,025+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "es06", "message": "closing ..." }
{"type": "server", "timestamp": "2019-09-02T13:00:23,036+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "es06", "message": "closed" }

Data Node new docker-compose.yml:

  - node.name=prafull
  - xpack.monitoring.enabled=false
  - node.master=false
  - node.data=true
  - node.ingest=false
  - cluster.name=data-nodes
  - discovery.seed_hosts=18.XX.XX.XX
  - bootstrap.memory_lock=true
  - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
  - transport.port=9300
4

This node cannot discover any of the other nodes in the cluster - it has only managed to discover itself. Unfortunately you've redacted the useful information from the logs (i.e. the IP addresses) but my best guess is that discovery.seed_hosts is configured incorrectly. It should contain the addresses of all the master-eligible nodes in the cluster.

5

Hi David,

As you have suggested I have added the list of all master eligible node in discovery.seed_hosts but still getting an error. Error, MasterNode, DataNode detail in the link below

Link https://gofile.io/?c=OudfnK
dataNode - https://gofile.io/?c=4wuRrn
MasterNode - https://gofile.io/?c=Baigtn

6

It still looks like a discovery problem, because this node can still only discover itself:

... have discovered [{es05}{HhMmBiBoTTaWzxfmEVpw2A}{klSdGaZZQFq9WbgpZqwdEw}{172.19.0.2}{172.19.0.2:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [172.31.1.160:9300, 172.18.0.2:9300, 172.18.0.3:9300, 172.18.0.4:9300, 172.18.0.5:9300] ...

Are you sure those are the right addresses? It looks like you're running in Docker, so maybe you have not configured the Docker network quite right.

If you're certain that this node should be able to connect to the other nodes at those addresses then try setting logger.org.elasticsearch.discovery: TRACE in the elasticsearch.yml file to get some more detailed information about where the discovery process is falling down.

7

Hi David,

Yes I am using docker and following the below structure

Docker 1:

  • 2 Master eligible node
  • 1 Master eligible and voting only node
  • 2 Coordinating node

Docker 2

  • 2 Data Node

abc

8

Hi David,

Thank You for your support, you were right I wasn't creating docker network properly.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.