java.io.StreamCorruptedException: invalid internal transport message format

Elastic Stack Elasticsearch
1

Hi all,
I have a three node ES cluster with IP (10.0.1.10,10.0.1.11,10.0.1.12), following WARN has been found in the log.
in the following log, a remote address has been mentioned which is not a member of cluster (10.0.1.244)

[2021-04-12T17:27:58,229][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [node-SRV110] exception caught on transport layer [NettyTcpChannel{localAddress=/10.0.1.10:9300, remoteAddress=/10.0.1.244:40297}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (74,1f,4a,84)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:167) [netty-handler-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1429) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:947) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-common-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-common-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
	at java.lang.Thread.run(Unknown Source) [?:1.8.0_152]
Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (74,1f,4a,84)
	at org.elasticsearch.transport.TcpTransport.validateMessageHeader(TcpTransport.java:1072) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:36) ~[transport-netty4-client-6.5.4.jar:6.5.4]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
	... 20 more
2

in addition,following message has been found in the log:

[2021-04-12T17:33:52,154][DEBUG][o.e.a.s.TransportSearchAction] [node-SRV110] [.monitoring-es-6-2021.04.06][0], node[6U240ajMSruuV3OR15F9hQ], [R], s[STARTED], a[id=b8Cv2w-5RFqjXFOasbtMEw]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[.monitoring-es-2-*, .monitoring-es-6-*], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=15, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, source={"size":0,"query":{"bool":{"filter":[{"term":{"cluster_uuid":{"value":"Cn7jg9X1TDGdYUNSFXp-nw","boost":1.0}}},{"term":{"source_node.uuid":{"value":"6U240ajMSruuV3OR15F9hQ","boost":1.0}}},{"term":{"source_node.uuid":{"value":"6U240ajMSruuV3OR15F9hQ","boost":1.0}}},{"range":{"timestamp":{"from":1618071376144,"to":1618074720717,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"check":{"date_histogram":{"field":"timestamp","interval":"NaNs","offset":0,"order":{"_key":"asc"},"keyed":false,"min_doc_count":0},"aggregations":{"metric":{"max":{"field":"node_stats.indices.segments.count"}}}}}}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [node-SRV111][10.0.1.11:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: failed to parse [NaNs]
	at org.elasticsearch.common.unit.TimeValue.parse(TimeValue.java:315) ~[elasticsearch-core-6.5.4.jar:6.5.4]
	at org.elasticsearch.common.unit.TimeValue.parseTimeValue(TimeValue.java:287) ~[elasticsearch-core-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.getIntervalAsTimeValue(DateHistogramAggregationBuilder.java:464) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.createRounding(DateHistogramAggregationBuilder.java:476) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.innerBuild(DateHistogramAggregationBuilder.java:429) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:311) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:37) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:336) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:807) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:616) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:592) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:367) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService.access$100(SearchService.java:121) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.4.jar:6.5.4]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:1.8.0_152]
	at java.lang.Thread.run(Unknown Source) [?:1.8.0_152]
Caused by: java.lang.NumberFormatException: For input string: "nan"
	at java.lang.NumberFormatException.forInputString(Unknown Source) ~[?:1.8.0_152]
	at java.lang.Long.parseLong(Unknown Source) ~[?:1.8.0_152]
	at java.lang.Long.parseLong(Unknown Source) ~[?:1.8.0_152]
	at org.elasticsearch.common.unit.TimeValue.parse(TimeValue.java:309) ~[elasticsearch-core-6.5.4.jar:6.5.4]
	at org.elasticsearch.common.unit.TimeValue.parseTimeValue(TimeValue.java:287) ~[elasticsearch-core-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.getIntervalAsTimeValue(DateHistogramAggregationBuilder.java:464) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.createRounding(DateHistogramAggregationBuilder.java:476) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.innerBuild(DateHistogramAggregationBuilder.java:429) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:311) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:37) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:336) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:807) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:616) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:592) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:367) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService.access$100(SearchService.java:121) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.5.4.jar:6.5.4]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.4.jar:6.5.4]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:1.8.0_152]
	at java.lang.Thread.run(Unknown Source) ~[?:1.8.0_152]

it is noted after some minutes of observing these messages, kibana dashboards are not accessible and "the request for this pannel failed. the aggregation key is missing from the response. check your permission for this request" message has been shown. which, after restarting kibana nodes, the issue will be resolved. Could you please advise me about these issues? Regards

3

It is noted that, observing "java.io.StreamCorruptedException: invalid internal transport message format" is followed by increasing client requests in kibana monitoring dashboard based on the following figures for nodes 10.0.1.10,10.0.1.11,10.0.1.12, respectively.

SRV110
SRV1101701×797 123 KB
SRV111
SRV1111686×783 117 KB
SRV112
SRV1121688×795 119 KB

4

What's the 10.0.1.244 host running?

Also please upgrade. 6.5 has been EOL for nearly a year now!

5

it seems this host sends many request to elasticsearch at some moments which leads kibana to lock accessing to it. is there any way of handling this issue without restarting kibana?

6

If it's not Kibana, then you shouldn't need to restart the Kibana process.

7

but, the issue will be resolved after restarting kibana

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.