io.netty.handler.ssl.SslHandshakeTimeoutException: handshake timed out after 10000ms

Elastic Stack Elasticsearch
1

Hi Team,
We are getting this below exception daily. so, can you please give solution for this exception.
we are using Elasticsearch 7.3.2, java version 1.8.0_131 and tomcat version 9.

2023-02-19 18:55:20.683 [elasticsearch[client][transport_worker][T#2]] WARN org.elasticsearch.transport.OutboundHandler - [103]:send message failed [channel: Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:port, remoteAddress=/ip:port, profile=default}]
io.netty.handler.ssl.SslHandshakeTimeoutException: handshake timed out after 10000ms
at io.netty.handler.ssl.SslHandler$5.run(SslHandler.java:2054) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
2023-02-19 18:55:20.685 [elasticsearch[client][transport_worker][T#2]] WARN org.elasticsearch.transport.OutboundHandler - [103]:sending transport message [Request{internal:tcp/handshake}{399342}{false}{false}{true}] of size [124] on [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:port, remoteAddress=/ip:port, profile=default}] took [10006ms] which is above the warn threshold of [5000ms] with success [false]

2

Welcome to our community! :smiley:

Please note that version is EOL and no longer supported, you should be looking to upgrade as a matter of urgency as it's very old.

Does this run your client code?

What is the output from the _cluster/stats?pretty&human API?

3

Elasticsearch 7.3 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

4

This is output of our cluster stats.
{
"_nodes": {
"total": 8,
"successful": 8,
"failed": 0
},
"cluster_name": "####",
"cluster_uuid": "DD00FjboRlmFLKVcdU2nPg",
"timestamp": 1677061175368,
"status": "green",
"indices": {
"count": 461,
"shards": {
"total": 4586,
"primaries": 2293,
"replication": 1.0,
"index": {
"shards": {
"min": 2,
"max": 10,
"avg": 9.947939262472884
},
"primaries": {
"min": 1,
"max": 5,
"avg": 4.973969631236442
},
"replication": {
"min": 1.0,
"max": 1.0,
"avg": 1.0
}
}
},
"docs": {
"count": 363072545,
"deleted": 25158248
},
"store": {
"size": "766.6gb",
"size_in_bytes": 823141903857
},
"fielddata": {
"memory_size": "106.3kb",
"memory_size_in_bytes": 108904,
"evictions": 0
},
"query_cache": {
"memory_size": "201.3mb",
"memory_size_in_bytes": 211109337,
"total_count": 32292900,
"hit_count": 7127915,
"miss_count": 25164985,
"cache_size": 3432,
"cache_count": 1101742,
"evictions": 1098310
},
"completion": {
"size": "0b",
"size_in_bytes": 0
},
"segments": {
"count": 12287,
"memory": "435mb",
"memory_in_bytes": 456229988,
"terms_memory": "178.8mb",
"terms_memory_in_bytes": 187565473,
"stored_fields_memory": "228.9mb",
"stored_fields_memory_in_bytes": 240045656,
"term_vectors_memory": "0b",
"term_vectors_memory_in_bytes": 0,
"norms_memory": "1.9mb",
"norms_memory_in_bytes": 2038336,
"points_memory": "14.3mb",
"points_memory_in_bytes": 15055183,
"doc_values_memory": "10.9mb",
"doc_values_memory_in_bytes": 11525340,
"index_writer_memory": "22.3mb",
"index_writer_memory_in_bytes": 23448406,
"version_map_memory": "1.6kb",
"version_map_memory_in_bytes": 1734,
"fixed_bit_set": "72.7mb",
"fixed_bit_set_memory_in_bytes": 76272424,
"max_unsafe_auto_id_timestamp": 1677059706968,
"file_sizes": {

  }
}

},
"nodes": {
"count": {
"total": 8,
"coordinating_only": 0,
"data": 5,
"ingest": 0,
"master": 3,
"voting_only": 0
},
"versions": [
"7.3.2"
],
"os": {
"available_processors": 108,
"allocated_processors": 108,
"names": [
{
"name": "Linux",
"count": 8
}
],
"pretty_names": [
{
"pretty_name": "Ubuntu 18.04.6 LTS",
"count": 8
}
],
"mem": {
"total": "848.9gb",
"total_in_bytes": 911572004864,
"free": "94.2gb",
"free_in_bytes": 101191860224,
"used": "754.7gb",
"used_in_bytes": 810380144640,
"free_percent": 11,
"used_percent": 89
}
},
"process": {
"cpu": {
"percent": 23
},
"open_file_descriptors": {
"min": 561,
"max": 35585,
"avg": 17628
}
},
"jvm": {
"max_uptime": "320d",
"max_uptime_in_millis": 27653202781,
"versions": [
{
"version": "1.8.0_131",
"vm_name": "Java HotSpot(TM) 64-Bit Server VM",
"vm_version": "25.131-b11",
"vm_vendor": "Oracle Corporation",
"bundled_jdk": true,
"using_bundled_jdk": false,
"count": 8
}
],
"mem": {
"heap_used": "35.3gb",
"heap_used_in_bytes": 37994793856,
"heap_max": "80.7gb",
"heap_max_in_bytes": 86701506560
},
"threads": 1064
},
"fs": {
"total": "2.8tb",
"total_in_bytes": 3166766153728,
"free": "1tb",
"free_in_bytes": 1119663431680,
"available": "1tb",
"available_in_bytes": 1119562768384
},
"plugins": [
{
"name": "repository-gcs",
"version": "7.3.2",
"elasticsearch_version": "7.3.2",
"java_version": "1.8",
"description": "The GCS repository plugin adds Google Cloud Storage support for repositories.",
"classname": "org.elasticsearch.repositories.gcs.GoogleCloudStoragePlugin",
"extended_plugins": [

    ],
    "has_native_controller": false
  }
],
"network_types": {
  "transport_types": {
    "security4": 8
  },
  "http_types": {
    "security4": 8
  }
},
"discovery_types": {
  "zen": 8
},
"packaging_types": [
  {
    "flavor": "default",
    "type": "tar",
    "count": 8
  }
]

}
}

5

In future please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

Other than the very old version that you need to upgrade ASAP, you look to be oversharded. You have nearly 600 shards per node with an average size of ~5.5GB, and with each node having ~10GB of heap assigned to it. That's going to be putting a lot of pressure on Elasticsearch to manage and is very inefficient.

You should look to increase the average size of your shards and to reduce your shard count.

6

overshards is the reason for a ssl.handshake timeout exception. But, we are getting this error recently.
This error causing because of creating overshards or something related to elastic?

7

Over sharding puts unnecessary load on the cluster that can impact connection time outs between nodes, yes.

You really need to upgrade first though, 7.3 is not supported.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.