# Marvel showing unresponsive nodes but active data **URL:** \ **Category:** Elasticsearch\ **Created:** [April 24, 2015, 6:26pm UTC](https://discuss.elastic.co/t/marvel-showing-unresponsive-nodes-but-active-data/23394 "2015-04-24T18:26:34Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Tristan\_Hammond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tristan_hammond/32/703_2.png) [@Tristan\_Hammond](https://discuss.elastic.co/u/Tristan_Hammond)\ **Post date:** [April 24, 2015, 6:26pm UTC](https://discuss.elastic.co/t/marvel-showing-unresponsive-nodes-but-active-data/23394/1 "2015-04-24T18:26:34Z")
Hi all. So we have an Elasticsearch cluster up and running. 5 nodes (1 master, 1 client and 3 data). At some point the Marvel dash began stating on all our nodes and indices that "no report has been received for more than 2m." I'm not sure when this seems to have started, but the cluster status is green, the shard and index count up top is correct, and the dash is showing correct document count, index rate and request rate on the indices. I've Googled around a bunch and haven't been able to find someone else having this issue. A rolling restart is something I'd considered, but since it's powering search on our site I'd like it to be a last resort. Any insight would be much appreciated. Cheers, - Tristan - -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/63760db3-2913-4f18-9442-9599187cbea3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/63760db3-2913-4f18-9442-9599187cbea3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tristan\_Hammond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tristan_hammond/32/703_2.png) [@Tristan\_Hammond](https://discuss.elastic.co/u/Tristan_Hammond)\ **Post date:** [April 30, 2015, 7:23pm UTC](https://discuss.elastic.co/t/marvel-showing-unresponsive-nodes-but-active-data/23394/2 "2015-04-30T19:23:53Z")
Hmm... Digging into this a little, it seems like it's an issue specifically w/ the Master node. If I load up Marvel on any of the other nodes, they see everything but the Master (and the updated index data it would hold). Cluster still reports green w/ all 5 nodes for some reason, though. It's only when I look at Marvel on the Master node that it looks like everything is unresponsive (yet still showing a green status and 5 nodes). I'll keep updating in case I figure out what's going on so other folks will have an answer if they come across this. Cheers, - Tristan - On Friday, April 24, 2015 at 1:26:35 PM UTC-5, Tristan Hammond wrote: > Hi all. > > So we have an Elasticsearch cluster up and running. 5 nodes (1 master, 1 > client and 3 data). At some point the Marvel dash began stating on all our > nodes and indices that "no report has been received for more than 2m." I'm > not sure when this seems to have started, but the cluster status is green, > the shard and index count up top is correct, and the dash is showing > correct document count, index rate and request rate on the indices. > > I've Googled around a bunch and haven't been able to find someone else > having this issue. A rolling restart is something I'd considered, but since > it's powering search on our site I'd like it to be a last resort. > > Any insight would be much appreciated. > > Cheers, > - Tristan - -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a2a44e4b-c40d-47b7-b822-6a9454cc3e95%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a2a44e4b-c40d-47b7-b822-6a9454cc3e95%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 30, 2015, 9:10pm UTC](https://discuss.elastic.co/t/marvel-showing-unresponsive-nodes-but-active-data/23394/3 "2015-04-30T21:10:01Z")
Check time is syncd between your nodes. On 1 May 2015 at 05:23, Tristan Hammond [tristan.hammond@gmail.com](mailto:tristan.hammond@gmail.com) wrote: > Hmm... > > Digging into this a little, it seems like it's an issue specifically w/ > the Master node. If I load up Marvel on any of the other nodes, they see > everything but the Master (and the updated index data it would hold). > Cluster still reports green w/ all 5 nodes for some reason, though. > > It's only when I look at Marvel on the Master node that it looks like > everything is unresponsive (yet still showing a green status and 5 nodes). > > I'll keep updating in case I figure out what's going on so other folks > will have an answer if they come across this. > > Cheers, > > - Tristan - > > On Friday, April 24, 2015 at 1:26:35 PM UTC-5, Tristan Hammond wrote: > > > Hi all. > > > > So we have an Elasticsearch cluster up and running. 5 nodes (1 master, 1 > > client and 3 data). At some point the Marvel dash began stating on all our > > nodes and indices that "no report has been received for more than 2m." I'm > > not sure when this seems to have started, but the cluster status is green, > > the shard and index count up top is correct, and the dash is showing > > correct document count, index rate and request rate on the indices. > > > > I've Googled around a bunch and haven't been able to find someone else > > having this issue. A rolling restart is something I'd considered, but since > > it's powering search on our site I'd like it to be a last resort. > > > > Any insight would be much appreciated. > > > > Cheers, > > - Tristan - > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/a2a44e4b-c40d-47b7-b822-6a9454cc3e95%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a2a44e4b-c40d-47b7-b822-6a9454cc3e95%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/a2a44e4b-c40d-47b7-b822-6a9454cc3e95%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a2a44e4b-c40d-47b7-b822-6a9454cc3e95%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9nrLHp1xanxN0m0Lhoc0MtjcoQNFNDXna2AYxB617pPg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9nrLHp1xanxN0m0Lhoc0MtjcoQNFNDXna2AYxB617pPg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tristan\_Hammond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tristan_hammond/32/703_2.png) [@Tristan\_Hammond](https://discuss.elastic.co/u/Tristan_Hammond)\ **Post date:** [May 15, 2015, 6:40pm UTC](https://discuss.elastic.co/t/marvel-showing-unresponsive-nodes-but-active-data/23394/4 "2015-05-15T18:40:54Z")
Mark, you're a lovely man. I finally got a spare few minutes to see what was up, and there was a 200s+ offset on the master. Not sure why our EC2 instance was not syncing time to correct that offset, as it's an AWS-built Ubuntu AMI. I corrected the offset on the master node and voilà! Things are back to normal now. So yea... if anyone else is experiencing weird misreporting for the master or any other node, check this is an easy thing to look at. Mark, again, thanks. Cheers, - Tristan - On Thursday, April 30, 2015 at 4:10:31 PM UTC-5, Mark Walkom wrote: > Check time is syncd between your nodes. > > On 1 May 2015 at 05:23, Tristan Hammond \<[tristan...@gmail.com](mailto:tristan...@gmail.com) > \\> wrote: > > > Hmm... > > > > Digging into this a little, it seems like it's an issue specifically w/ > > the Master node. If I load up Marvel on any of the other nodes, they see > > everything but the Master (and the updated index data it would hold). > > Cluster still reports green w/ all 5 nodes for some reason, though. > > > > It's only when I look at Marvel on the Master node that it looks like > > everything is unresponsive (yet still showing a green status and 5 nodes). > > > > I'll keep updating in case I figure out what's going on so other folks > > will have an answer if they come across this. > > > > Cheers, > > > > - Tristan - > > > > On Friday, April 24, 2015 at 1:26:35 PM UTC-5, Tristan Hammond wrote: > > > > > Hi all. > > > > > > So we have an Elasticsearch cluster up and running. 5 nodes (1 master, 1 > > > client and 3 data). At some point the Marvel dash began stating on all our > > > nodes and indices that "no report has been received for more than 2m." I'm > > > not sure when this seems to have started, but the cluster status is green, > > > the shard and index count up top is correct, and the dash is showing > > > correct document count, index rate and request rate on the indices. > > > > > > I've Googled around a bunch and haven't been able to find someone else > > > having this issue. A rolling restart is something I'd considered, but since > > > it's powering search on our site I'd like it to be a last resort. > > > > > > Any insight would be much appreciated. > > > > > > Cheers, > > > - Tristan - > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/a2a44e4b-c40d-47b7-b822-6a9454cc3e95%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a2a44e4b-c40d-47b7-b822-6a9454cc3e95%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/a2a44e4b-c40d-47b7-b822-6a9454cc3e95%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a2a44e4b-c40d-47b7-b822-6a9454cc3e95%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2a492903-cb12-4c48-8bdb-1999920c81b2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2a492903-cb12-4c48-8bdb-1999920c81b2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:13am UTC](https://discuss.elastic.co/t/marvel-showing-unresponsive-nodes-but-active-data/23394/5 "2017-07-06T00:13:49Z")