# Cluster overview empty, no indexes, connecting to old node name - not even showing itself in list **URL:** \ **Category:** Elasticsearch\ **Created:** [November 6, 2013, 9:16am UTC](https://discuss.elastic.co/t/cluster-overview-empty-no-indexes-connecting-to-old-node-name-not-even-showing-itself-in-list/14285 "2013-11-06T09:16:02Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![RNELord](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rnelord/32/2002_2.png) [@RNELord](https://discuss.elastic.co/u/RNELord)\ **Post date:** [November 6, 2013, 9:16am UTC](https://discuss.elastic.co/t/cluster-overview-empty-no-indexes-connecting-to-old-node-name-not-even-showing-itself-in-list/14285/1 "2013-11-06T09:16:02Z")
Hey everyone, Last night we had elasticsearch screw up big time in production. Let me describe the issue: - Via /\_plugin/head the cluster overview was completely empty - it wasn't even showing itself in the list of nodes. - Obviously, no indexes were showing, however in the data browser, it WAS showing indexes existed - In cluster status, all shards were showing on itself, however were showing as 'unassigned' Restarting the service on one of the nodes did not help - instead the logs showed it trying to connect to itself as master under its previous node name, not its new node name. This then failed saying it wasn't a master. Both nodes were not talking to eachother and doing this exact same thing, trying to elect themselves as master and failing. The head plugin was available and loading, however cluster state was showing as 'not connected'. It wasn't until we shut down the services on both nodes, added two fresh nodes in sequence (giving the first time to elect itself as master), adding 2nd, them talking, then continuing do bring up the original 2 nodes that it started showing all the nodes in the cluster instead of a completely empty cluster state. We are using the EC2 security groups plugin running on EC2 instances using version 0.90.2 with no other plugins installed - sematext is installed and running in standalone mode talking to the JVM instead of being loaded via a plugin. Using openjdk 7. Has anyone else encountered this issue where it seems the clustering layer completely fails? Is this a known bug? I can't see any reference of this issue in the mailing list yet. I'd attach cluster output but since it was production and sites were down, in our haste to fix it we didn't dump any health statuses. It's also worth mentioning that the logs were completely void of any information when the nodes stopped serving the indexes with previous log entry being about 24 hours prior about a failed search. Thanks to anyone who can help or lead us in the right direction of what happened. We're also about to start digging through to do an RCA. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [November 6, 2013, 6:31pm UTC](https://discuss.elastic.co/t/cluster-overview-empty-no-indexes-connecting-to-old-node-name-not-even-showing-itself-in-list/14285/2 "2013-11-06T18:31:08Z")
My first thought is that you encountered a split brain scenario, but to have head show a complete empty cluster is strange. With split brain, you will at least see part of the data. Have you tried to debug with the es2unix utility? ( [GitHub - elastic/es2unix: Command-line ES](https://github.com/elasticsearch/es2unix) ) Use the lifecycle command to understand what the master status was during the failure. I personally do not care for the openjdk JVM. Too many issues in the past, but it has been awhile since I have used it. Cheers, Ivan On Wed, Nov 6, 2013 at 1:16 AM, RNELord [rnelord@gmail.com](mailto:rnelord@gmail.com) wrote: > Hey everyone, > > Last night we had elasticsearch screw up big time in production. Let me > describe the issue: > > - Via /\_plugin/head the cluster overview was completely empty - it wasn't > even showing itself in the list of nodes. > - Obviously, no indexes were showing, however in the data browser, it WAS > showing indexes existed > - In cluster status, all shards were showing on itself, however were > showing as 'unassigned' > > Restarting the service on one of the nodes did not help - instead the logs > showed it trying to connect to itself as master under its previous node > name, not its new node name. This then failed saying it wasn't a master. > Both nodes were not talking to eachother and doing this exact same thing, > trying to elect themselves as master and failing. The head plugin was > available and loading, however cluster state was showing as 'not connected'. > > It wasn't until we shut down the services on both nodes, added two fresh > nodes in sequence (giving the first time to elect itself as master), adding > 2nd, them talking, then continuing do bring up the original 2 nodes that it > started showing all the nodes in the cluster instead of a completely empty > cluster state. > > We are using the EC2 security groups plugin running on EC2 instances using > version 0.90.2 with no other plugins installed - sematext is installed and > running in standalone mode talking to the JVM instead of being loaded via a > plugin. Using openjdk 7. > > Has anyone else encountered this issue where it seems the clustering layer > completely fails? Is this a known bug? I can't see any reference of this > issue in the mailing list yet. > > I'd attach cluster output but since it was production and sites were down, > in our haste to fix it we didn't dump any health statuses. It's also worth > mentioning that the logs were completely void of any information when the > nodes stopped serving the indexes with previous log entry being about 24 > hours prior about a failed search. > > Thanks to anyone who can help or lead us in the right direction of what > happened. We're also about to start digging through to do an RCA. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:08am UTC](https://discuss.elastic.co/t/cluster-overview-empty-no-indexes-connecting-to-old-node-name-not-even-showing-itself-in-list/14285/3 "2017-07-06T02:08:29Z")