# HELP! My production is dead **URL:** \ **Category:** Elasticsearch\ **Created:** [July 1, 2012, 2:24am UTC](https://discuss.elastic.co/t/help-my-production-is-dead/8271 "2012-07-01T02:24:01Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Eugene\_Strokin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eugene_strokin/32/1238_2.png) [@Eugene\_Strokin](https://discuss.elastic.co/u/Eugene_Strokin)\ **Post date:** [July 1, 2012, 2:24am UTC](https://discuss.elastic.co/t/help-my-production-is-dead/8271/1 "2012-07-01T02:24:01Z")
Hello, Suddenly, ES start being unresponsive from time to time. I've decided to restart it. But it stack at initializing: [2012-06-30 21:01:49,080][INFO][bootstrap] max\_open\_files [29977] [2012-06-30 21:01:49,089][INFO][node] [Piledriver] {0.19.0}[6554]: initializing ... [2012-06-30 21:01:49,093][INFO][plugins] [Piledriver] loaded [], sites [] I'm waiting for 20 minutes for initializing complete. This is relatively small index (around 1 GB). It shouldn't take that much time. I see that Java process takes some CPU, but not 100% (20-40 % in average). What it could be? Should I start to upload my back ups? Thank you ---
**Author:** ![pedro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pedro/32/2564_2.png) [@pedro](https://discuss.elastic.co/u/pedro)\ **Post date:** [July 1, 2012, 2:30am UTC](https://discuss.elastic.co/t/help-my-production-is-dead/8271/2 "2012-07-01T02:30:00Z")
/etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; /etc/init.d/ntp start Stupid leap second On Sun Jul 1 03:24:01 2012, Eugene Strokin wrote: > Hello, > Suddenly, ES start being unresponsive from time to time. > I've decided to restart it. > But it stack at initializing: > [2012-06-30 21:01:49,080][INFO][bootstrap] max\_open\_files [29977] > [2012-06-30 21:01:49,089][INFO][node] [Piledriver] {0.19.0}[6554]: > initializing ... > [2012-06-30 21:01:49,093][INFO][plugins] [Piledriver] loaded , > sites > > I'm waiting for 20 minutes for initializing complete. This is > relatively small index (around 1 GB). It shouldn't take that much time. > > I see that Java process takes some CPU, but not 100% (20-40 % in average). > > What it could be? > Should I start to upload my back ups? > > Thank you ---
**Author:** ![Eugene\_Strokin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eugene_strokin/32/1238_2.png) [@Eugene\_Strokin](https://discuss.elastic.co/u/Eugene_Strokin)\ **Post date:** [July 1, 2012, 2:32am UTC](https://discuss.elastic.co/t/help-my-production-is-dead/8271/3 "2012-07-01T02:32:13Z")
Thank you Pedro, but what does it mean really? How it helps? On Saturday, June 30, 2012 10:30:00 PM UTC-4, Pedro Alves wrote: > /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; > /etc/init.d/ntp start > > Stupid leap second > > On Sun Jul 1 03:24:01 2012, Eugene Strokin wrote: > > > Hello, > > Suddenly, ES start being unresponsive from time to time. > > I've decided to restart it. > > But it stack at initializing: > > [2012-06-30 21:01:49,080][INFO][bootstrap] max\_open\_files [29977] > > [2012-06-30 21:01:49,089][INFO][node] [Piledriver] {0.19.0}[6554]: > > initializing ... > > [2012-06-30 21:01:49,093][INFO][plugins] [Piledriver] loaded , > > sites > > > > I'm waiting for 20 minutes for initializing complete. This is > > relatively small index (around 1 GB). It shouldn't take that much time. > > > > I see that Java process takes some CPU, but not 100% (20-40 % in > > average). > > > > What it could be? > > Should I start to upload my back ups? > > > > Thank you ---
**Author:** ![pedro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pedro/32/2564_2.png) [@pedro](https://discuss.elastic.co/u/pedro)\ **Post date:** [July 1, 2012, 2:41am UTC](https://discuss.elastic.co/t/help-my-production-is-dead/8271/4 "2012-07-01T02:41:57Z")
Who cares as long as it works? 😛 On Sun Jul 1 03:32:13 2012, Eugene Strokin wrote: > Thank you Pedro, > but what does it mean really? > How it helps? > > On Saturday, June 30, 2012 10:30:00 PM UTC-4, Pedro Alves wrote: > > ``` > /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; > /etc/init.d/ntp start > > Stupid leap second > > On Sun Jul 1 03:24:01 2012, Eugene Strokin wrote: > > Hello, > > Suddenly, ES start being unresponsive from time to time. > > I've decided to restart it. > > But it stack at initializing: > > [2012-06-30 21:01:49,080][INFO][bootstrap] max_open_files [29977] > > [2012-06-30 21:01:49,089][INFO][node] [Piledriver] > {0.19.0}[6554]: > > initializing ... > > [2012-06-30 21:01:49,093][INFO][plugins] [Piledriver] loaded [], > > sites [] > > > > I'm waiting for 20 minutes for initializing complete. This is > > relatively small index (around 1 GB). It shouldn't take that > much time. > > > > I see that Java process takes some CPU, but not 100% (20-40 % in > average). > > > > What it could be? > > Should I start to upload my back ups? > > > > Thank you > > > > ``` ---
**Author:** ![Eugene\_Strokin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eugene_strokin/32/1238_2.png) [@Eugene\_Strokin](https://discuss.elastic.co/u/Eugene_Strokin)\ **Post date:** [July 1, 2012, 2:56am UTC](https://discuss.elastic.co/t/help-my-production-is-dead/8271/5 "2012-07-01T02:56:59Z")
Hmm,.. I don't know if this helped, but after about 30 minutes ES got started and it works now... My system doesn't have /etc/init.d/ntp file, so I doubt the command was the reason why it's started. I'm actualy very surprised it took so much time. Could something be done about this? I also noticed that there are about 1700 files for node 0 of the index. Other nodes have something around 200 files. I guess this is wrong, but I'm not sure about this. Could it be fixed somehow? Or is this wrong at all? Anyway, if someone had such problems, please share. Any information is helpful. Thank you, Eugene S. On Saturday, June 30, 2012 10:41:57 PM UTC-4, Pedro Alves wrote: > Who cares as long as it works? 😛 > > On Sun Jul 1 03:32:13 2012, Eugene Strokin wrote: > > > Thank you Pedro, > > but what does it mean really? > > How it helps? > > > > On Saturday, June 30, 2012 10:30:00 PM UTC-4, Pedro Alves wrote: > > > > ``` > > /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; > > /etc/init.d/ntp start > > > > Stupid leap second > > > > On Sun Jul 1 03:24:01 2012, Eugene Strokin wrote: > > > Hello, > > > Suddenly, ES start being unresponsive from time to time. > > > I've decided to restart it. > > > But it stack at initializing: > > > [2012-06-30 21:01:49,080][INFO][bootstrap] max_open_files > > > > ``` > > [29977] > > > ``` > > > [2012-06-30 21:01:49,089][INFO][node] [Piledriver] > > {0.19.0}[6554]: > > > initializing ... > > > [2012-06-30 21:01:49,093][INFO][plugins] [Piledriver] loaded [], > > > sites [] > > > > > > I'm waiting for 20 minutes for initializing complete. This is > > > relatively small index (around 1 GB). It shouldn't take that > > much time. > > > > > > I see that Java process takes some CPU, but not 100% (20-40 % in > > average). > > > > > > What it could be? > > > Should I start to upload my back ups? > > > > > > Thank you > > > > > > > ``` ---
**Author:** ![dostrow](https://avatars.discourse-cdn.com/v4/letter/d/d6d6ee/32.png) [@dostrow](https://discuss.elastic.co/u/dostrow)\ **Post date:** [July 1, 2012, 3:34am UTC](https://discuss.elastic.co/t/help-my-production-is-dead/8271/6 "2012-07-01T03:34:52Z")
On Saturday, June 30, 2012 7:30:00 PM UTC-7, Pedro Alves wrote: > /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; > /etc/init.d/ntp start > > Stupid leap second THANK YOU!!! This just fixed my cluster than went down today, I would NEVER have thought of this. The cluster was taking literally hours to restart the nodes, run this, everything restarts right away. WTF... ---
**Author:** ![Eugene\_Strokin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eugene_strokin/32/1238_2.png) [@Eugene\_Strokin](https://discuss.elastic.co/u/Eugene_Strokin)\ **Post date:** [July 2, 2012, 6:17pm UTC](https://discuss.elastic.co/t/help-my-production-is-dead/8271/7 "2012-07-02T18:17:02Z")
Apparently it was the problem: [http://www.google.com/hostednews/afp/article/ALeqM5iW5Bq-w6vhpZZR2XItR4EBJbbxPw?docId=CNG.b993298106a8bbaafb124651a0577fd2.1d1](http://www.google.com/hostednews/afp/article/ALeqM5iW5Bq-w6vhpZZR2XItR4EBJbbxPw?docId=CNG.b993298106a8bbaafb124651a0577fd2.1d1) Still not clear why it caused the problem with long start up time, but as Pedro said, as long as it works))) On Saturday, June 30, 2012 11:34:52 PM UTC-4, dostrow wrote: > On Saturday, June 30, 2012 7:30:00 PM UTC-7, Pedro Alves wrote: > > > /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; > > /etc/init.d/ntp start > > > > Stupid leap second > > THANK YOU!!! This just fixed my cluster than went down today, I would > NEVER have thought of this. The cluster was taking literally hours to > restart the nodes, run this, everything restarts right away. WTF... ---
**Author:** ![llowder](https://avatars.discourse-cdn.com/v4/letter/l/e79b87/32.png) [@llowder](https://discuss.elastic.co/u/llowder)\ **Post date:** [July 2, 2012, 7:02pm UTC](https://discuss.elastic.co/t/help-my-production-is-dead/8271/8 "2012-07-02T19:02:30Z")
Leap seconds caused a LOT of problems over the weekend. Long story short (if I understand things right), there is a bug in ntpd that causes the CPU to lock anytime anything needs a timestamp when there is an 'active' leapsecond. The fix is either to stop ntpd before the leapsecond is applied, the commands mentioned, or a server restart. Sometimes, a combination of these is needed. On Monday, July 2, 2012 1:17:02 PM UTC-5, Eugene Strokin wrote: > Apparently it was the problem: > > [http://www.google.com/hostednews/afp/article/ALeqM5iW5Bq-w6vhpZZR2XItR4EBJbbxPw?docId=CNG.b993298106a8bbaafb124651a0577fd2.1d1](http://www.google.com/hostednews/afp/article/ALeqM5iW5Bq-w6vhpZZR2XItR4EBJbbxPw?docId=CNG.b993298106a8bbaafb124651a0577fd2.1d1) > Still not clear why it caused the problem with long start up time, but as > Pedro said, as long as it works))) > > On Saturday, June 30, 2012 11:34:52 PM UTC-4, dostrow wrote: > > > On Saturday, June 30, 2012 7:30:00 PM UTC-7, Pedro Alves wrote: > > > > > /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; > > > /etc/init.d/ntp start > > > > > > Stupid leap second > > > > THANK YOU!!! This just fixed my cluster than went down today, I would > > NEVER have thought of this. The cluster was taking literally hours to > > restart the nodes, run this, everything restarts right away. WTF... ---
**Author:** ![Pavel\_Dvorak](https://avatars.discourse-cdn.com/v4/letter/p/b9e5f3/32.png) [@Pavel\_Dvorak](https://discuss.elastic.co/u/Pavel_Dvorak)\ **Post date:** [July 4, 2012, 8:08pm UTC](https://discuss.elastic.co/t/help-my-production-is-dead/8271/9 "2012-07-04T20:08:16Z")
My cluster also crashed few days ago so it was probably this. It was weird, it stopped respoding so I restarted it. It restarted fine and it seems no data were lost. On Monday, July 2, 2012 8:17:02 PM UTC+2, Eugene Strokin wrote: > Apparently it was the problem: > > [http://www.google.com/hostednews/afp/article/ALeqM5iW5Bq-w6vhpZZR2XItR4EBJbbxPw?docId=CNG.b993298106a8bbaafb124651a0577fd2.1d1](http://www.google.com/hostednews/afp/article/ALeqM5iW5Bq-w6vhpZZR2XItR4EBJbbxPw?docId=CNG.b993298106a8bbaafb124651a0577fd2.1d1) > Still not clear why it caused the problem with long start up time, but as > Pedro said, as long as it works))) > > On Saturday, June 30, 2012 11:34:52 PM UTC-4, dostrow wrote: > > > On Saturday, June 30, 2012 7:30:00 PM UTC-7, Pedro Alves wrote: > > > > > /etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date; > > > /etc/init.d/ntp start > > > > > > Stupid leap second > > > > THANK YOU!!! This just fixed my cluster than went down today, I would > > NEVER have thought of this. The cluster was taking literally hours to > > restart the nodes, run this, everything restarts right away. WTF... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:21am UTC](https://discuss.elastic.co/t/help-my-production-is-dead/8271/10 "2017-07-06T03:21:28Z")