# 2AM to 3AM connectivity issues **URL:** \ **Category:** Elasticsearch\ **Created:** [October 25, 2013, 6:51pm UTC](https://discuss.elastic.co/t/2am-to-3am-connectivity-issues/14120 "2013-10-25T18:51:27Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Bryan\_Murphy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bryan_murphy/32/2023_2.png) [@Bryan\_Murphy](https://discuss.elastic.co/u/Bryan_Murphy)\ **Post date:** [October 25, 2013, 6:51pm UTC](https://discuss.elastic.co/t/2am-to-3am-connectivity-issues/14120/1 "2013-10-25T18:51:27Z")
We have a simple elastic search cluster (two nodes + logstash). Every night at around 2am if we have 4-5 or more indexes in the database (each index takes up roughly 1gb) elastic search stops responding to queries and our pagers start going off. An hour or two later everything clears up and the monitoring system is happy again. I'd like an uninterrupted nights sleep. Does anybody have an idea what could be going on during this period? As far as I can tell the other parts of this service (logstash+redis queues) aren't misbehaving. It very consistently happens between 2am and 3am when nobody is lucent or available. Thanks, Bryan -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 26, 2013, 6:41am UTC](https://discuss.elastic.co/t/2am-to-3am-connectivity-issues/14120/2 "2013-10-26T06:41:57Z")
What do the ES logs from that time tell you? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 26 October 2013 05:51, Bryan Murphy [bmurphy1976@gmail.com](mailto:bmurphy1976@gmail.com) wrote: > We have a simple Elasticsearch cluster (two nodes + logstash). Every > night at around 2am if we have 4-5 or more indexes in the database (each > index takes up roughly 1gb) Elasticsearch stops responding to queries and > our pagers start going off. > > An hour or two later everything clears up and the monitoring system is > happy again. > > I'd like an uninterrupted nights sleep. Does anybody have an idea what > could be going on during this period? As far as I can tell the other parts > of this service (logstash+redis queues) aren't misbehaving. > > It very consistently happens between 2am and 3am when nobody is lucent or > available. > > Thanks, > Bryan > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![vineeth\_mohan\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vineeth_mohan_2/32/747_2.png) [@vineeth\_mohan\_2](https://discuss.elastic.co/u/vineeth_mohan_2)\ **Post date:** [October 26, 2013, 7:07am UTC](https://discuss.elastic.co/t/2am-to-3am-connectivity-issues/14120/3 "2013-10-26T07:07:46Z")
Quite interesting problem . Some years back, i had a similar problem. During morning 8 AM , things don't work as it usually had to.People even thought that the machines were haunted 😛 . So what happened was that , there was a script in the cron which tipped off at that time and it did the backlog indexing for the entire day at that particular time. So the ghost was actually a cron doing all its backlogs at 8 AM. See if the issues are similar. Thanks Vineeth On Sat, Oct 26, 2013 at 12:11 PM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)wrote: > What do the ES logs from that time tell you? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 26 October 2013 05:51, Bryan Murphy [bmurphy1976@gmail.com](mailto:bmurphy1976@gmail.com) wrote: > > > We have a simple Elasticsearch cluster (two nodes + logstash). Every > > night at around 2am if we have 4-5 or more indexes in the database (each > > index takes up roughly 1gb) Elasticsearch stops responding to queries and > > our pagers start going off. > > > > An hour or two later everything clears up and the monitoring system is > > happy again. > > > > I'd like an uninterrupted nights sleep. Does anybody have an idea what > > could be going on during this period? As far as I can tell the other parts > > of this service (logstash+redis queues) aren't misbehaving. > > > > It very consistently happens between 2am and 3am when nobody is lucent or > > available. > > > > Thanks, > > Bryan > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [October 26, 2013, 8:04pm UTC](https://discuss.elastic.co/t/2am-to-3am-connectivity-issues/14120/4 "2013-10-26T20:04:30Z")
Bryan, you didn't mention which timezone you are in. Is it 2am CET by any chance? Are there any messages in the log files around 2am? If you don't see anything in log files try enabling slow index and slow query log and if it will wake you up another time, try running hot\_thread[http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html) a few times around 2:30 am and post results here. On Saturday, October 26, 2013 3:07:46 AM UTC-4, vineeth mohan wrote: > Quite interesting problem . > > Some years back, i had a similar problem. During morning 8 AM , things > don't work as it usually had to.People even thought that the machines were > haunted 😛 . > So what happened was that , there was a script in the cron which tipped > off at that time and it did the backlog indexing for the entire day at that > particular time. > So the ghost was actually a cron doing all its backlogs at 8 AM. > See if the issues are similar. > > Thanks > Vineeth > > On Sat, Oct 26, 2013 at 12:11 PM, Mark Walkom \<[ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)\ > > > wrote: > > > What do the ES logs from that time tell you? > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 26 October 2013 05:51, Bryan Murphy \<[bmurp...@gmail.com](mailto:bmurp...@gmail.com) \\>wrote: > > > > > We have a simple Elasticsearch cluster (two nodes + logstash). Every > > > night at around 2am if we have 4-5 or more indexes in the database (each > > > index takes up roughly 1gb) Elasticsearch stops responding to queries and > > > our pagers start going off. > > > > > > An hour or two later everything clears up and the monitoring system is > > > happy again. > > > > > > I'd like an uninterrupted nights sleep. Does anybody have an idea what > > > could be going on during this period? As far as I can tell the other parts > > > of this service (logstash+redis queues) aren't misbehaving. > > > > > > It very consistently happens between 2am and 3am when nobody is lucent > > > or available. > > > > > > Thanks, > > > Bryan > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bryan\_Murphy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bryan_murphy/32/2023_2.png) [@Bryan\_Murphy](https://discuss.elastic.co/u/Bryan_Murphy)\ **Post date:** [October 26, 2013, 8:53pm UTC](https://discuss.elastic.co/t/2am-to-3am-connectivity-issues/14120/5 "2013-10-26T20:53:13Z")
Central Time Zone. I'll look into those options. As far as I can tell nothing else is going on on that machine at during that time but I'll definitely take a closer look. It's been a tough one because of the time it happens (and our desire to get back to sleep). Thanks! Bryan On Sat, Oct 26, 2013 at 3:04 PM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote: > Bryan, you didn't mention which timezone you are in. Is it 2am CET by any > chance? Are there any messages in the log files around 2am? If you don't > see anything in log files try enabling slow index and slow query log and if > it will wake you up another time, try running hot\_thread[http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html) a > few times around 2:30 am and post results here. > > On Saturday, October 26, 2013 3:07:46 AM UTC-4, vineeth mohan wrote: > > > Quite interesting problem . > > > > Some years back, i had a similar problem. During morning 8 AM , things > > don't work as it usually had to.People even thought that the machines were > > haunted 😛 . > > So what happened was that , there was a script in the cron which tipped > > off at that time and it did the backlog indexing for the entire day at that > > particular time. > > So the ghost was actually a cron doing all its backlogs at 8 AM. > > See if the issues are similar. > > > > Thanks > > Vineeth > > > > On Sat, Oct 26, 2013 at 12:11 PM, Mark Walkom [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)wrote: > > > > > What do the ES logs from that time tell you? > > > > > > Regards, > > > Mark Walkom > > > > > > Infrastructure Engineer > > > Campaign Monitor > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > > > On 26 October 2013 05:51, Bryan Murphy [bmurp...@gmail.com](mailto:bmurp...@gmail.com) wrote: > > > > > > > We have a simple Elasticsearch cluster (two nodes + logstash). Every > > > > night at around 2am if we have 4-5 or more indexes in the database (each > > > > index takes up roughly 1gb) Elasticsearch stops responding to queries and > > > > our pagers start going off. > > > > > > > > An hour or two later everything clears up and the monitoring system is > > > > happy again. > > > > > > > > I'd like an uninterrupted nights sleep. Does anybody have an idea what > > > > could be going on during this period? As far as I can tell the other parts > > > > of this service (logstash+redis queues) aren't misbehaving. > > > > > > > > It very consistently happens between 2am and 3am when nobody is lucent > > > > or available. > > > > > > > > Thanks, > > > > Bryan > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > . > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:10am UTC](https://discuss.elastic.co/t/2am-to-3am-connectivity-issues/14120/6 "2017-07-06T02:10:27Z")