# java.io.StreamCorruptedException **URL:** \ **Category:** Elasticsearch\ **Created:** [August 30, 2019, 9:36am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception/197485 "2019-08-30T09:36:56Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Prafull](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prafull/32/56422_2.png) [@Prafull](https://discuss.elastic.co/u/Prafull)\ **Post date:** [August 30, 2019, 9:36am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception/197485/1 "2019-08-30T09:36:56Z")
Hi , When I am trying to connect DataNode with Master Node Getting error below as written below: ``` {"type": "server", "timestamp": "2019-08-30T12:08:21,959+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "prafull_singh", "message": "master not discovered yet: have discovered [{prafull_singh}{shKCd54aQFef7Vt6kMNkLQ}{J6YhGHQwTsCfnuF_c2WWyg}{172.18.0.2}{172.18.0.2:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [34.244.248.50:9200] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0" } ``` prafull | {"type": "server", "timestamp": "2019-08-30T12:08:29,972+0000", "level": "WARN", "component": "o.e.t.TcpTransport", "cluster.name": "docker-cluster", "node.name": "prafull\_singh", "message": "exception caught on transport layer [Netty4TcpChannel{localAddress=/172.18.0.2:41170, remoteAddress=/34.XXX.XXX.XXX:9200}], closing connection" , prafull | "stacktrace": ["io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)", prafull | "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar:4.1.36.Final]", prafull | "at java.lang.Thread.run(Thread.java:835) [?:?]", prafull | "Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)", prafull | "at org.elasticsearch.transport.TcpTransport.readHeaderBuffer(TcpTransport.java:745) ~[elasticsearch-7.3.1.jar:7.3.1]", prafull | "at org.elasticsearch.transport.TcpTransport.readMessageLength(TcpTransport.java:731) ~[elasticsearch-7.3.1.jar:7.3.1]", prafull | "at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:40) ~[transport-netty4-client-7.3.1.jar:7.3.1]", prafull | "at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]", prafull | "at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final]", prafull | "... 20 more"] } prafull | {"type": "server", "timestamp": "2019-08-30T12:08:29,981+0000", "level": "WARN", "component": "o.e.t.TcpTransport", "cluster.name": "docker-cluster", "node.name": "prafull\_singh", "message": "exception caught on transport layer [Netty4TcpChannel{localAddress=/172.18.0.2:41170, remoteAddress=/34.XXX.XXX.XXX:9200}], closing connection" , prafull | "stacktrace": ["io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)", prafull | "at **dataNode docker-compose.yml** version: '2.2' services: prafull: image: [docker.elastic.co/elasticsearch/elasticsearch:7.3.1](http://docker.elastic.co/elasticsearch/elasticsearch:7.3.1) container\_name: prafull environment: - node.name=prafull\_singh - xpack.monitoring.enabled=false - node.master=false - node.data=true - node.ingest=false - cluster.name=docker-cluster - discovery.seed\_hosts=34.XX.XX.XX:9200 - bootstrap.memory\_lock=true - "ES\_JAVA\_OPTS=-Xms254m -Xmx254m" ulimits: memlock: soft: -1 hard: -1 volumes: - prafull:/usr/share/elasticsearch/data networks: - esnet volumes: prafull: driver: local networks: esnet: ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 30, 2019, 4:53pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception/197485/2 "2019-08-30T16:53:05Z")
Elastic communicated internally on port 9300 and you have configured port 9200 for discovery.seed\_hosts. I also suspect you will need to increase the size of the heap to at least the double in order to be able to do anything useful. ---
**Author:** ![Prafull](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prafull/32/56422_2.png) [@Prafull](https://discuss.elastic.co/u/Prafull)\ **Post date:** [September 2, 2019, 6:18am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception/197485/3 "2019-09-02T06:18:09Z")
Hi Christian, After updating both **port and heap size** , data node is up but not able to join master nodes cluster. When I am running `curl -X GET "IP:9200/_cat/master"` it gave me proper master node detail. Find Error Detai Below {"type": "server", "timestamp": "2019-09-02T12:59:05,824+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine\_memory=4135084032, xpack.installed=true, ml.max\_open\_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" } {"type": "server", "timestamp": "2019-09-02T12:59:15,825+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine\_memory=4135084032, xpack.installed=true, ml.max\_open\_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" } {"type": "server", "timestamp": "2019-09-02T12:59:25,825+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine\_memory=4135084032, xpack.installed=true, ml.max\_open\_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" } {"type": "server", "timestamp": "2019-09-02T12:59:35,826+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine\_memory=4135084032, xpack.installed=true, ml.max\_open\_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" } {"type": "server", "timestamp": "2019-09-02T12:59:45,827+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine\_memory=4135084032, xpack.installed=true, ml.max\_open\_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" } {"type": "server", "timestamp": "2019-09-02T12:59:55,828+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine\_memory=4135084032, xpack.installed=true, ml.max\_open\_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" } {"type": "server", "timestamp": "2019-09-02T13:00:05,829+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine\_memory=4135084032, xpack.installed=true, ml.max\_open\_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" } {"type": "server", "timestamp": "2019-09-02T13:00:15,830+0000", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "docker-cluster", "node.name": "es06", "message": "master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine\_memory=4135084032, xpack.installed=true, ml.max\_open\_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0" } {"type": "server", "timestamp": "2019-09-02T13:00:19,217+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "es06", "message": "stopping ..." } {"type": "server", "timestamp": "2019-09-02T13:00:19,238+0000", "level": "INFO", "component": "o.e.x.w.WatcherService", "cluster.name": "docker-cluster", "node.name": "es06", "message": "stopping watch service, reason [shutdown initiated]" } {"type": "server", "timestamp": "2019-09-02T13:00:19,348+0000", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "docker-cluster", "node.name": "es06", "message": "[controller/102] [Main.cc@150] Ml controller exiting" } {"type": "server", "timestamp": "2019-09-02T13:00:19,349+0000", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "docker-cluster", "node.name": "es06", "message": "Native controller process has stopped - no new native processes can be started" } {"type": "server", "timestamp": "2019-09-02T13:00:23,025+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "es06", "message": "stopped" } {"type": "server", "timestamp": "2019-09-02T13:00:23,025+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "es06", "message": "closing ..." } {"type": "server", "timestamp": "2019-09-02T13:00:23,036+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "es06", "message": "closed" } **Data Node new docker-compose.yml:** ``` - node.name=prafull - xpack.monitoring.enabled=false - node.master=false - node.data=true - node.ingest=false - cluster.name=data-nodes - discovery.seed_hosts=18.XX.XX.XX - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - transport.port=9300 ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 2, 2019, 2:51pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception/197485/4 "2019-09-02T14:51:36Z")
> [@Prafull](#): > > `master not discovered yet: have discovered [{es06}{ut6HwH0yTyi31lcMzxyGFQ}{mZ1vNIAjTSes2StkLOI2sA}{XX.XX.XX.XX}{XX.XX.XX.XX:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [XX.XX.XX.XX:9300] from hosts providers` This node cannot discover any of the other nodes in the cluster - it has only managed to discover itself. Unfortunately you've redacted the useful information from the logs (i.e. the IP addresses) but my best guess is that `discovery.seed_hosts` is configured incorrectly. It should contain the addresses of all the master-eligible nodes in the cluster. ---
**Author:** ![Prafull](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prafull/32/56422_2.png) [@Prafull](https://discuss.elastic.co/u/Prafull)\ **Post date:** [September 3, 2019, 6:45am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception/197485/5 "2019-09-03T06:45:08Z")
Hi David, As you have suggested I have added the list of all master eligible node in **discovery.seed\_hosts** but still getting an error. Error, MasterNode, DataNode detail in the link below **Link** [https://gofile.io/?c=OudfnK](https://gofile.io/?c=OudfnK) dataNode - [https://gofile.io/?c=4wuRrn](https://gofile.io/?c=4wuRrn) MasterNode - [https://gofile.io/?c=Baigtn](https://gofile.io/?c=Baigtn) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 3, 2019, 8:54am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception/197485/6 "2019-09-03T08:54:06Z")
It still looks like a discovery problem, because this node can still only discover itself: `... have discovered [{es05}{HhMmBiBoTTaWzxfmEVpw2A}{klSdGaZZQFq9WbgpZqwdEw}{172.19.0.2}{172.19.0.2:9300}{d}{ml.machine_memory=4135084032, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [172.31.1.160:9300, 172.18.0.2:9300, 172.18.0.3:9300, 172.18.0.4:9300, 172.18.0.5:9300] ...` Are you sure those are the right addresses? It looks like you're running in Docker, so maybe you have not configured the Docker network quite right. If you're certain that this node should be able to connect to the other nodes at those addresses then try setting `logger.org.elasticsearch.discovery: TRACE` in the `elasticsearch.yml` file to get some more detailed information about where the discovery process is falling down. ---
**Author:** ![Prafull](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prafull/32/56422_2.png) [@Prafull](https://discuss.elastic.co/u/Prafull)\ **Post date:** [September 3, 2019, 11:15am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception/197485/7 "2019-09-03T11:15:07Z")
Hi David, Yes I am using docker and following the below structure **Docker 1:** - 2 Master eligible node - 1 Master eligible and voting only node - 2 Coordinating node **Docker 2** - 2 Data Node ![abc](https://us1.discourse-cdn.com/elastic/original/3X/0/4/0419696d0728dffd244806fb92da4f8593dbdbbd.png) ---
**Author:** ![Prafull](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prafull/32/56422_2.png) [@Prafull](https://discuss.elastic.co/u/Prafull)\ **Post date:** [September 4, 2019, 8:13am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception/197485/8 "2019-09-04T08:13:30Z")
Hi David, Thank You for your support, you were right I wasn't creating docker network properly. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 2, 2019, 8:19am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception/197485/9 "2019-10-02T08:19:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.