[o.e.t.TransportService] Received response for a request that has timed out

Elastic Stack Elasticsearch
1

We are receiving the following error

{ElasticsearchLogger} [o.e.t.TransportService] Received response for a request that has timed out, sent [21.3s/21361ms] ago, timed out [5.6s/5682ms] ago, action [indices:monitor/stats[n]], node  {--XX--}{--XX--}{--XX--}{127.0.0.1}{127.0.0.1:9201}{dim}{xpack.installed=true, transform.node=false}], id [3787]

{ElasticsearchLogger} [o.e.c.InternalClusterInfoService] failed to retrieve shard stats from node [--XX--]: [--XX--][127.0.0.1:9201][indices:monitor/stats[n]] request_id [4022] timed out after [15043ms]

{ElasticsearchLogger} [o.e.t.TransportService] Received response for a request that has timed out, sent [42.2s/42281ms] ago, timed out [27.2s/27238ms] ago, action [indices:monitor/stats[n]], node [{--XX--}{--XX--}{--XX--}{127.0.0.1}{127.0.0.1:9201}{dim}{xpack.installed=true, transform.node=false}], id [4022]

{ElasticsearchLogger} [o.e.m.f.FsHealthService] health check of [--XX--] took [9271ms] which is above the warn threshold of [5s]

{ElasticsearchLogger} [o.e.c.InternalClusterInfoService] failed to retrieve shard stats from node [--XX--]: [--XX--][127.0.0.1:9201][indices:monitor/stats[n]] request_id [4355] timed out after [14925ms]

We tried to increase the JVM heap size from 8GB to 16GB
But still facing the issue.

Need suggestions how we can resolve this issue

2

What is the output of:

GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v
3

Here are the results:
GET /

9200
9200733×580 100 KB

GET /_cat/health?v

health_V
health_V1573×273 69.9 KB

GET /_cat/indices?v

Indices_V
Indices_V1429×410 124 KB

GET /_cat/nodes?v

nodes_v
nodes_v1085×297 57.2 KB

4

Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

5

What's the heap size?

Could you upgrade your cluster to the latest 7.17 version?

6

What's the heap size?

16 GB

Could you upgrade your cluster to the latest 7.17 version?

It would not be possible to upgrade to newer version

Please do let me know if you have any lead on what might be reason for such exception

7

Do you have another node running on the machine? What else is running there?

8

1. Do you have another node running on the machine?

No.

2. What else is running there?

Enterprise Vault 14.2.2 

ElasticSearch 

Veritas Cluster Service

Symantec Endpoint protection

Veritas NBU Agent 

Microsoft Office 2016 

Axon Agent 

Dell SecureWorks
9

In your logs I can see: [127.0.0.1:9201] but in the screen capture, I can see localhost:9200... So I'm confused... What is running under port 9201?

10

In our case the ports are select randomly either 9200 or 9201 when we restart our service. Its a single node cluster.

11

I disagree. There's something in the logs of the 9200 which mentions a call from 9201.
Meaning that 2 nodes are running on the same machine.

It's super unclear what you are doing TBH.

12

What is happening in our Environment

The logs that were provided were form different time and the snapshots of GET API are from different time.
So basically what is happening is when we restart are internal services a port is selected randomly through which further communication takes place.
After the logs were provided ours services were restarted, because of the random selection of the ports the port over which the communication is taking place changed to 9200.
If we restart it again it might either change the port to 9200 or 9201.
Its either or.

So lets say current point of time 9200 is selected, The logs will show 9200 and The GET API will be successful only for 9200.
and similarly if 9201 port is select then logs will show 9201 and the GET API will be successful only for 9201

So basically when the logs were collected 9201 was set but because of service restart the port changed and was set to 9200.

I hope this helps.

13

Elasticsearch uses by default 9200 and does not pick a random port.
It uses 9201 if AND ONLY IF something is already running in 9200.

But let's come back to the discussion.

I don't see anything wrong on Elasticsearch side. But Elasticsearch should run alone on a server. All the other services you listed below might steal RAM/CPU/...

  • Enterprise Vault 14.2.2
  • Veritas Cluster Service
  • Symantec Endpoint protection
  • Veritas NBU Agent
  • Microsoft Office 2016
  • Axon Agent
  • Dell SecureWorks

So if it's a production machine, please remove all the other services.
If it's a dev machine, I think you can ignore the warnings.

14

Is there a way to disable these warning.
Any registry key or environment variable or config change or anything else that can used to disable these calls

15

Is it a production machine?

16

Yes, Its a production machine

17

As i said:

Make sure that Elasticsearch starts only on 9200 port and that you never try to start Elasticsearch multiple times on the same machine (which by default is not possible).

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.