# java.io.StreamCorruptedException: invalid internal transport message format **URL:** \ **Category:** Elasticsearch\ **Created:** [April 12, 2021, 3:31pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format/269899 "2021-04-12T15:31:20Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![sahere37](https://avatars.discourse-cdn.com/v4/letter/s/b2d939/32.png) [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Post date:** [April 12, 2021, 3:31pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format/269899/1 "2021-04-12T15:31:20Z")
Hi all, I have a three node ES cluster with IP (10.0.1.10,10.0.1.11,10.0.1.12), following WARN has been found in the log. in the following log, a remote address has been mentioned which is not a member of cluster (10.0.1.244) ``` [2021-04-12T17:27:58,229][WARN][o.e.x.s.t.n.SecurityNetty4ServerTransport] [node-SRV110] exception caught on transport layer [NettyTcpChannel{localAddress=/10.0.1.10:9300, remoteAddress=/10.0.1.244:40297}], closing connection io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (74,1f,4a,84) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:167) [netty-handler-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.30.Final.jar:4.1.30.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-common-4.1.30.Final.jar:4.1.30.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-transport-4.1.30.Final.jar:4.1.30.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final] at java.lang.Thread.run(Unknown Source) [?:1.8.0_152] Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (74,1f,4a,84) at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1072) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:36) ~[transport-netty4-client-6.5.4.jar:6.5.4] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final] ... 20 more ``` ---
**Author:** ![sahere37](https://avatars.discourse-cdn.com/v4/letter/s/b2d939/32.png) [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Post date:** [April 12, 2021, 3:31pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format/269899/2 "2021-04-12T15:31:43Z")
in addition,following message has been found in the log: ``` [2021-04-12T17:33:52,154][DEBUG][o.e.a.s.TransportSearchAction] [node-SRV110] [.monitoring-es-6-2021.04.06][0], node[6U240ajMSruuV3OR15F9hQ], [R], s[STARTED], a[id=b8Cv2w-5RFqjXFOasbtMEw]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.monitoring-es-2-*, .monitoring-es-6-*], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=15, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, source={"size":0,"query":{"bool":{"filter":[{"term":{"cluster_uuid":{"value":"Cn7jg9X1TDGdYUNSFXp-nw","boost":1.0}}},{"term":{"source_node.uuid":{"value":"6U240ajMSruuV3OR15F9hQ","boost":1.0}}},{"term":{"source_node.uuid":{"value":"6U240ajMSruuV3OR15F9hQ","boost":1.0}}},{"range":{"timestamp":{"from":1618071376144,"to":1618074720717,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"check":{"date_histogram":{"field":"timestamp","interval":"NaNs","offset":0,"order":{"_key":"asc"},"keyed":false,"min_doc_count":0},"aggregations":{"metric":{"max":{"field":"node_stats.indices.segments.count"}}}}}}}] lastShard [true] org.elasticsearch.transport.RemoteTransportException: [node-SRV111][10.0.1.11:9300][indices:data/read/search[phase/query]] Caused by: java.lang.IllegalArgumentException: failed to parse [NaNs] at org.elasticsearch.common.unit.TimeValue.parse(TimeValue.java:315) ~[elasticsearch-core-6.5.4.jar:6.5.4] at org.elasticsearch.common.unit.TimeValue.parseTimeValue(TimeValue.java:287) ~[elasticsearch-core-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.getIntervalAsTimeValue(DateHistogramAggregationBuilder.java:464) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.createRounding(DateHistogramAggregationBuilder.java:476) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.innerBuild(DateHistogramAggregationBuilder.java:429) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:311) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:37) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:336) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService.parseSource(SearchService.java:807) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService.createContext(SearchService.java:616) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:592) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:367) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService.access$100(SearchService.java:121) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.4.jar:6.5.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:1.8.0_152] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:1.8.0_152] at java.lang.Thread.run(Unknown Source) [?:1.8.0_152] Caused by: java.lang.NumberFormatException: For input string: "nan" at java.lang.NumberFormatException.forInputString(Unknown Source) ~[?:1.8.0_152] at java.lang.Long.parseLong(Unknown Source) ~[?:1.8.0_152] at java.lang.Long.parseLong(Unknown Source) ~[?:1.8.0_152] at org.elasticsearch.common.unit.TimeValue.parse(TimeValue.java:309) ~[elasticsearch-core-6.5.4.jar:6.5.4] at org.elasticsearch.common.unit.TimeValue.parseTimeValue(TimeValue.java:287) ~[elasticsearch-core-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.getIntervalAsTimeValue(DateHistogramAggregationBuilder.java:464) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.createRounding(DateHistogramAggregationBuilder.java:476) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.innerBuild(DateHistogramAggregationBuilder.java:429) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:311) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:37) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:336) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService.parseSource(SearchService.java:807) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService.createContext(SearchService.java:616) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:592) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:367) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService.access$100(SearchService.java:121) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.5.4.jar:6.5.4] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.4.jar:6.5.4] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:1.8.0_152] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:1.8.0_152] at java.lang.Thread.run(Unknown Source) ~[?:1.8.0_152] ``` it is noted after some minutes of observing these messages, kibana dashboards are not accessible and "`the request for this pannel failed. the aggregation key is missing from the response. check your permission for this request`" message has been shown. which, after restarting kibana nodes, the issue will be resolved. Could you please advise me about these issues? Regards ---
**Author:** ![sahere37](https://avatars.discourse-cdn.com/v4/letter/s/b2d939/32.png) [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Post date:** [April 12, 2021, 3:53pm UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format/269899/3 "2021-04-12T15:53:26Z")
It is noted that, observing "java.io.StreamCorruptedException: invalid internal transport message format" is followed by increasing client requests in kibana monitoring dashboard based on the following figures for nodes 10.0.1.10,10.0.1.11,10.0.1.12, respectively. ![SRV110](https://us1.discourse-cdn.com/elastic/original/3X/c/c/cc617ce716a9cbbd444c45d7fdb7ea4f23c99f2c.jpeg) ![SRV111](https://us1.discourse-cdn.com/elastic/original/3X/6/5/65dfbe16fba996669b1f8cc50205c1f8f4f82da6.jpeg) ![SRV112](https://us1.discourse-cdn.com/elastic/original/3X/5/f/5f7ac4ea4fdb461be2eeeb61cc3cccf77687a3e0.jpeg) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 14, 2021, 12:23am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format/269899/4 "2021-04-14T00:23:21Z")
> [@sahere37](#): > > `10.0.1.244` What's the `10.0.1.244` host running? > [@sahere37](#): > > `elasticsearch-core-6.5.4` Also please upgrade. 6.5 has been [EOL](https://www.elastic.co/support/eol) for nearly a year now! ---
**Author:** ![sahere37](https://avatars.discourse-cdn.com/v4/letter/s/b2d939/32.png) [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Post date:** [April 14, 2021, 9:44am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format/269899/5 "2021-04-14T09:44:46Z")
it seems this host sends many request to elasticsearch at some moments which leads kibana to lock accessing to it. is there any way of handling this issue without restarting kibana? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 14, 2021, 10:04am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format/269899/6 "2021-04-14T10:04:27Z")
If it's not Kibana, then you shouldn't need to restart the Kibana process. ---
**Author:** ![sahere37](https://avatars.discourse-cdn.com/v4/letter/s/b2d939/32.png) [@sahere37](https://discuss.elastic.co/u/sahere37)\ **Post date:** [April 24, 2021, 9:21am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format/269899/7 "2021-04-24T09:21:45Z")
but, the issue will be resolved after restarting kibana ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 22, 2021, 9:21am UTC](https://discuss.elastic.co/t/java-io-streamcorruptedexception-invalid-internal-transport-message-format/269899/8 "2021-05-22T09:21:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.