# org.elasticsearch.transport.ReceiveTimeoutTransportException...timed out after \[5002ms\] without any hints and exceptions in cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [December 9, 2014, 2:48pm UTC](https://discuss.elastic.co/t/org-elasticsearch-transport-receivetimeouttransportexception-timed-out-after-5002ms-without-any-hints-and-exceptions-in-cluster/21165 "2014-12-09T14:48:10Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![deer](https://avatars.discourse-cdn.com/v4/letter/d/f19dbf/32.png) [@deer](https://discuss.elastic.co/u/deer)\ **Post date:** [December 9, 2014, 2:48pm UTC](https://discuss.elastic.co/t/org-elasticsearch-transport-receivetimeouttransportexception-timed-out-after-5002ms-without-any-hints-and-exceptions-in-cluster/21165/1 "2014-12-09T14:48:10Z")
Hi, My elasticsearch cluster recently has a high response time. Elasticsearch cluster (version 1.1.1 + number of node : 8) Java client (client.transport.sniff is true) After checking, I found that all java clients have the exception org.elasticsearch.transport.ReceiveTimeoutTransportException...timed out after [5002ms] it only happens to the connection to the same node(the problem node). Master node and other nodes are fine. The cluster doing fine after removing the problem node. I've enabled the DEBUG level in the logging.yml and checked all the log in master and nodes. there is no exception and hints for the timeout. Any other options to trace this error? The nodes in the cluster can see each others. There should be no problem for the zen discovery. I also try to use telnet in the client machine to connect the problem node. It is working. in the problem node, we found the log of dmesg : possible syn flooding on port 9300. sending cookies Fixing the net.ipv4.tcp\_max\_syn\_backlog does not help. Reboot the machine and re-install the elasticsearch do not help. I try to use the problem node to host a cluster by itself and use the same java client to connect it. It is working fine. There is not exception and stack trace in both server side and client side when client side has the exception : org.elasticsearch.transport.ReceiveTimeoutTransportException...timed out after [5002ms] Not sure is it related to network issue / machine issue / elasticsearch issue. What can I do to know more about this timeout exception? Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1a211bd9-9d8f-46a1-a311-384912ed8da8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1a211bd9-9d8f-46a1-a311-384912ed8da8%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 9, 2014, 2:57pm UTC](https://discuss.elastic.co/t/org-elasticsearch-transport-receivetimeouttransportexception-timed-out-after-5002ms-without-any-hints-and-exceptions-in-cluster/21165/2 "2014-12-09T14:57:01Z")
Nodes timing out can be indicative of heavy GC. Do the logs show anything in that regards? Can you share more info on how big your nodes are, what your dataset size is, what java version you are on? On 9 December 2014 at 15:48, Hui [dannyhui1103@gmail.com](mailto:dannyhui1103@gmail.com) wrote: > Hi, > > My elasticsearch cluster recently has a high response time. > > Elasticsearch cluster (version 1.1.1 + number of node : 8) > Java client (client.transport.sniff is true) > > After checking, I found that all java clients have the exception > org.elasticsearch.transport.ReceiveTimeoutTransportException...timed out > after [5002ms] > it only happens to the connection to the same node(the problem node). > > Master node and other nodes are fine. The cluster doing fine after > removing the problem node. > > I've enabled the DEBUG level in the logging.yml and checked all the log in > master and nodes. there is no exception and hints for the timeout. > Any other options to trace this error? > > The nodes in the cluster can see each others. There should be no problem > for the zen discovery. > I also try to use telnet in the client machine to connect the problem > node. It is working. > > in the problem node, > we found the log of dmesg : possible syn flooding on port 9300. sending > cookies > Fixing the net.ipv4.tcp\_max\_syn\_backlog does not help. > > Reboot the machine and re-install the elasticsearch do not help. > > I try to use the problem node to host a cluster by itself and use the same > java client to connect it. It is working fine. > > There is not exception and stack trace in both server side and client side > when client side has the exception : > org.elasticsearch.transport.ReceiveTimeoutTransportException...timed out > after [5002ms] > > Not sure is it related to network issue / machine issue / elasticsearch > issue. What can I do to know more about this timeout exception? > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1a211bd9-9d8f-46a1-a311-384912ed8da8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1a211bd9-9d8f-46a1-a311-384912ed8da8%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/1a211bd9-9d8f-46a1-a311-384912ed8da8%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1a211bd9-9d8f-46a1-a311-384912ed8da8%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X\_eVs54ayu8GJ2sFF%3Dx%3DaDcwM8HyiLcFawrME4X5CAXJA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_eVs54ayu8GJ2sFF%3Dx%3DaDcwM8HyiLcFawrME4X5CAXJA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![deer](https://avatars.discourse-cdn.com/v4/letter/d/f19dbf/32.png) [@deer](https://discuss.elastic.co/u/deer)\ **Post date:** [December 10, 2014, 12:52am UTC](https://discuss.elastic.co/t/org-elasticsearch-transport-receivetimeouttransportexception-timed-out-after-5002ms-without-any-hints-and-exceptions-in-cluster/21165/3 "2014-12-10T00:52:08Z")
Hi, Thanks for the reply. Checked they are not doing any GC in all nodes. For cluster : - version 1.1.1 - 8 nodes - there are 2 indexs. index 1 size: 55.1G (164G) docs: 4,272,425 (7,155,663) index 2 size: 113G (341G) docs: 7,717,476 (11,271,866) For each nodes : memory 14gb Both server and client side are using jdk1.6.0\_30 average request count : ~1000 average response time : ~100ms Only single node has the timeout issue. Detail log(Client) : 2014-12-09 08:36:30,959 INFO org.elasticsearch.client.transport - [Alkhema] failed to get local cluster state for [10.1.4.196:9200][vs5uD2kLTXGkWDNrgsAZig][cluster\_name][inet[/10.1.4.196:9200]], disconnecting... org.elasticsearch.transport.ReceiveTimeoutTransportException: [10.1.4.196:9200][inet[/10.1.4.196:9300]][cluster/state] request\_id [1074667] timed out after [5002ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:356) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) No exceptions, gc and log in server side when we have the timeout issue. Many thanks. On Tuesday, December 9, 2014 10:57:28 PM UTC+8, Mark Walkom wrote: > Nodes timing out can be indicative of heavy GC. Do the logs show anything > in that regards? > > Can you share more info on how big your nodes are, what your dataset size > is, what java version you are on? > > On 9 December 2014 at 15:48, Hui \<[dannyh...@gmail.com](mailto:dannyh...@gmail.com) \\> > wrote: > > > Hi, > > > > My elasticsearch cluster recently has a high response time. > > > > Elasticsearch cluster (version 1.1.1 + number of node : 8) > > Java client (client.transport.sniff is true) > > > > After checking, I found that all java clients have the exception > > org.elasticsearch.transport.ReceiveTimeoutTransportException...timed out > > after [5002ms] > > it only happens to the connection to the same node(the problem node). > > > > Master node and other nodes are fine. The cluster doing fine after > > removing the problem node. > > > > I've enabled the DEBUG level in the logging.yml and checked all the log > > in master and nodes. there is no exception and hints for the timeout. > > Any other options to trace this error? > > > > The nodes in the cluster can see each others. There should be no problem > > for the zen discovery. > > I also try to use telnet in the client machine to connect the problem > > node. It is working. > > > > in the problem node, > > we found the log of dmesg : possible syn flooding on port 9300. sending > > cookies > > Fixing the net.ipv4.tcp\_max\_syn\_backlog does not help. > > > > Reboot the machine and re-install the elasticsearch do not help. > > > > I try to use the problem node to host a cluster by itself and use the > > same java client to connect it. It is working fine. > > > > There is not exception and stack trace in both server side and client > > side when client side has the exception : > > org.elasticsearch.transport.ReceiveTimeoutTransportException...timed out > > after [5002ms] > > > > Not sure is it related to network issue / machine issue / elasticsearch > > issue. What can I do to know more about this timeout exception? > > > > Thanks! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/1a211bd9-9d8f-46a1-a311-384912ed8da8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1a211bd9-9d8f-46a1-a311-384912ed8da8%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/1a211bd9-9d8f-46a1-a311-384912ed8da8%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1a211bd9-9d8f-46a1-a311-384912ed8da8%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3f43d657-b482-4cbd-8b5c-3f580356cd69%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3f43d657-b482-4cbd-8b5c-3f580356cd69%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:44am UTC](https://discuss.elastic.co/t/org-elasticsearch-transport-receivetimeouttransportexception-timed-out-after-5002ms-without-any-hints-and-exceptions-in-cluster/21165/4 "2017-07-06T00:44:36Z")