# Failed to execute on node while the cluster running about 20 minutes later **URL:** \ **Category:** Elasticsearch\ **Created:** [January 3, 2019, 7:08am UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-while-the-cluster-running-about-20-minutes-later/162751 "2019-01-03T07:08:43Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![liming8519](https://avatars.discourse-cdn.com/v4/letter/l/ea666f/32.png) [@liming8519](https://discuss.elastic.co/u/liming8519)\ **Post date:** [January 3, 2019, 7:08am UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-while-the-cluster-running-about-20-minutes-later/162751/1 "2019-01-03T07:08:43Z")
Hi, i'm new using Elastic, I'm facing the issue that my Elasticsearch master nodes got ReceiveTimeoutTransportException after running about 20 minutes. Elasticsearch version - 6.3.2 Ram - 8gb Heap size - 4gb Documents - 10(max) Java 1.8 OS Ubuntu Server 18.04 VM in Azure **This is config** ##### master-node-1 cluster.name: de-test network.host: 0.0.0.0 discovery.zen.ping.unicast.hosts: ["104.41.161.123:9300","13.78.36.124:9300","40.115.154.30:9300"] discovery.zen.minimum\_master\_nodes: 2 node.name: master-node-1 ##### master-node-2 cluster.name: de-test network.host: 0.0.0.0 discovery.zen.ping.unicast.hosts: ["104.41.161.123:9300","13.78.36.124:9300","40.115.154.30:9300"] discovery.zen.minimum\_master\_nodes: 2 node.name: master-node-2 ##### master-node-3 cluster.name: de-test network.host: 0.0.0.0 discovery.zen.ping.unicast.hosts: ["104.41.161.123:9300","13.78.36.124:9300","40.115.154.30:9300"] discovery.zen.minimum\_master\_nodes: 2 node.name: master-node-3 ###### network interface of master-node-1: **eth0:** inet 172.20.0.4 netmask 255.255.255.0 broadcast 172.20.0.4 **eth0:0:** inet 104.41.161.123 netmask 255.255.255.255 broadcast 104.41.161.123 **This is error** -\> ``` ...... [2019-01-03T06:32:29,164][INFO][o.e.c.s.MasterService] [master-node-2] zen-disco-elected-as-master ([1] nodes joined)[,], reason: new_master {master-node-2}{-b8ZPb9DTcarvOYfHdgtHg}{2gVzr0YSQBGl9GcPW6XRug}{40.115.154.30}{40.115.154.30:9300}{ml.machine_memory=8365367296, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, added {{master-node-1}{PUGB8is1SJm9XEeRJ6bVjA}{xzjwGTefRSaxINpyi4_-Qw}{104.41.161.123}{104.41.161.123:9300}{ml.machine_memory=8365367296, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},} [2019-01-03T06:32:29,245][INFO][o.e.c.s.ClusterApplierService] [master-node-2] new_master {master-node-2}{-b8ZPb9DTcarvOYfHdgtHg}{2gVzr0YSQBGl9GcPW6XRug}{40.115.154.30}{40.115.154.30:9300}{ml.machine_memory=8365367296, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, added {{master-node-1}{PUGB8is1SJm9XEeRJ6bVjA}{xzjwGTefRSaxINpyi4_-Qw}{104.41.161.123}{104.41.161.123:9300}{ml.machine_memory=8365367296, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {master-node-2}{-b8ZPb9DTcarvOYfHdgtHg}{2gVzr0YSQBGl9GcPW6XRug}{40.115.154.30}{40.115.154.30:9300}{ml.machine_memory=8365367296, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([1] nodes joined)[,]]]) ...... [2019-01-03T06:32:33,515][INFO][o.e.c.r.a.AllocationService] [master-node-2] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[sg6-auditlog-2019.01.03][2]] ...]). [2019-01-03T06:43:14,485][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [master-node-2] failed to execute on node [VANTBBkPR_qZsNUCWttXGA] org.elasticsearch.transport.ReceiveTimeoutTransportException: [master-node-3][13.78.36.124:9300][cluster:monitor/nodes/stats[n]] request_id [1515] timed out after [15000ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:979) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [elasticsearch-6.3.0.jar:6.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191] [2019-01-03T06:43:14,485][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [master-node-2] failed to execute on node [PUGB8is1SJm9XEeRJ6bVjA] org.elasticsearch.transport.ReceiveTimeoutTransportException: [master-node-1][104.41.161.123:9300][cluster:monitor/nodes/stats[n]] request_id [1514] timed out after [15000ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:979) [elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [elasticsearch-6.3.0.jar:6.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191] ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 3, 2019, 9:04am UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-while-the-cluster-running-about-20-minutes-later/162751/2 "2019-01-03T09:04:58Z")
I reckon it's about 10 minutes between starting up your cluster and the first error: ```nohighlight [2019-01-03T06:32:33,515][INFO][o.e.c.r.a.AllocationService] [master-node-2] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[sg6-auditlog-2019.01.03][2]] ...]). [2019-01-03T06:43:14,485][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [master-node-2] failed to execute on node [VANTBBkPR_qZsNUCWttXGA] ``` (please use the `` button to format logs, it makes them much easier to read) This suggests a connectivity issue - see the note in the manual about [long-lived connections](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/modules-transport.html#_long_lived_idle_connections). Are you running on GCP? There is an idle connection timeout of [10 minutes by default](https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-instances) there. ---
**Author:** ![liming8519](https://avatars.discourse-cdn.com/v4/letter/l/ea666f/32.png) [@liming8519](https://discuss.elastic.co/u/liming8519)\ **Post date:** [January 3, 2019, 9:11am UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-while-the-cluster-running-about-20-minutes-later/162751/3 "2019-01-03T09:11:18Z")
Sorry for the format, I'm running on Azure without firewall. This is sysctl ``` ming@mix-search-de-master-1:~$ sudo sysctl -p kernel.sysrq = 1 vm.max_map_count = 655360 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_intvl = 60 net.ipv4.tcp_keepalive_probes = 20 ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 3, 2019, 9:35am UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-while-the-cluster-running-about-20-minutes-later/162751/4 "2019-01-03T09:35:18Z")
It seems that [Azure's default timeout is 4 minutes](https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-tcp-idle-timeout). Your keep alive time is 600 seconds, i.e. 10 minutes. I think you need to set it much lower in such a network. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 31, 2019, 9:35am UTC](https://discuss.elastic.co/t/failed-to-execute-on-node-while-the-cluster-running-about-20-minutes-later/162751/5 "2019-01-31T09:35:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.