# Master node keeps crashing **URL:** \ **Category:** Elasticsearch\ **Created:** [June 10, 2014, 7:16am UTC](https://discuss.elastic.co/t/master-node-keeps-crashing/18007 "2014-06-10T07:16:38Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Gaurav\_Arora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gaurav_arora/32/1492_2.png) [@Gaurav\_Arora](https://discuss.elastic.co/u/Gaurav_Arora)\ **Post date:** [June 10, 2014, 7:16am UTC](https://discuss.elastic.co/t/master-node-keeps-crashing/18007/1 "2014-06-10T07:16:38Z")
I'm having a very odd problem with one of my elasticsearch clusters. The master node on the cluster crashes randomly. The cluster is running on 3 different ec2 instances. My cluster configuration is: 3 nodes (1 master, all data nodes) 600 GB of data (3k IOPS EBS volumes) 700 million documents There are no hprof files anywhere so I don't think this was a memory problem. I have set the log level to DEBUG but there is no information in them at all. I'm not sure where to start debugging this problem. I realise my post is not very helpful but if I could get pointers on where to start I will produce logs. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 10, 2014, 7:23am UTC](https://discuss.elastic.co/t/master-node-keeps-crashing/18007/2 "2014-06-10T07:23:01Z")
How much RAM per node, what java flavour and version, what ES version? Are the logs showing any OOM? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 10 June 2014 17:16, Gaurav Arora [gauravsworld@gmail.com](mailto:gauravsworld@gmail.com) wrote: > I'm having a very odd problem with one of my elasticsearch clusters. The > master node on the cluster crashes randomly. The cluster is running on 3 > different ec2 instances. > > My cluster configuration is: > 3 nodes (1 master, all data nodes) > 600 GB of data (3k IOPS EBS volumes) > 700 million documents > > There are no hprof files anywhere so I don't think this was a memory > problem. I have set the log level to DEBUG but there is no information in > them at all. > > I'm not sure where to start debugging this problem. I realise my post is > not very helpful but if I could get pointers on where to start I will > produce logs. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624aVupm0sUJTi7u-z7%2Bw1tNA2ogy0iE6PoQ%3DRQFx7Xfy%3DA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624aVupm0sUJTi7u-z7%2Bw1tNA2ogy0iE6PoQ%3DRQFx7Xfy%3DA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Gaurav\_Arora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gaurav_arora/32/1492_2.png) [@Gaurav\_Arora](https://discuss.elastic.co/u/Gaurav_Arora)\ **Post date:** [June 10, 2014, 7:36am UTC](https://discuss.elastic.co/t/master-node-keeps-crashing/18007/3 "2014-06-10T07:36:35Z")
I am using the latest openjdk version 7 installed from ubuntu repos. ubuntu@es1:~$ java -version java version "1.7.0\_51" OpenJDK Runtime Environment (IcedTea 2.4.6) (7u51-2.4.6-1ubuntu4) OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode) ES is set to run with -Xms14075m -Xmx14075m with bootstrap.mlockall set to true. The instance is a r3.large instance. There is no OOM in the logs anywhere, the last line of the log shows that an index was created about 15-20 minutes before the crash. Thats it. On Tuesday, June 10, 2014 12:53:41 PM UTC+5:30, Mark Walkom wrote: > How much RAM per node, what java flavour and version, what ES version? > > Are the logs showing any OOM? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 10 June 2014 17:16, Gaurav Arora \<[gaurav...@gmail.com](mailto:gaurav...@gmail.com) \\> > wrote: > > > I'm having a very odd problem with one of my elasticsearch clusters. The > > master node on the cluster crashes randomly. The cluster is running on 3 > > different ec2 instances. > > > > My cluster configuration is: > > 3 nodes (1 master, all data nodes) > > 600 GB of data (3k IOPS EBS volumes) > > 700 million documents > > > > There are no hprof files anywhere so I don't think this was a memory > > problem. I have set the log level to DEBUG but there is no information in > > them at all. > > > > I'm not sure where to start debugging this problem. I realise my post is > > not very helpful but if I could get pointers on where to start I will > > produce logs. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/541fc112-fafd-49c7-9a89-bac70fac298d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/541fc112-fafd-49c7-9a89-bac70fac298d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![sirkubax](https://avatars.discourse-cdn.com/v4/letter/s/f0a364/32.png) [@sirkubax](https://discuss.elastic.co/u/sirkubax)\ **Post date:** [June 10, 2014, 9:20am UTC](https://discuss.elastic.co/t/master-node-keeps-crashing/18007/4 "2014-06-10T09:20:20Z")
curl -XGET '[http://localhost:9200/\_nodes/\_all/process?pretty=true](http://localhost:9200/_nodes/_all/process?pretty=true)' |less Have you been considering "max\_file\_descriptors"? W dniu wtorek, 10 czerwca 2014 09:36:35 UTC+2 użytkownik Gaurav Arora napisał: > I am using the latest openjdk version 7 installed from ubuntu repos. > > ubuntu@es1:~$ java -version > java version "1.7.0\_51" > OpenJDK Runtime Environment (IcedTea 2.4.6) (7u51-2.4.6-1ubuntu4) > OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode) > > ES is set to run with -Xms14075m -Xmx14075m with bootstrap.mlockall set to > true. > > The instance is a r3.large instance. > > There is no OOM in the logs anywhere, the last line of the log shows that > an index was created about 15-20 minutes before the crash. Thats it. > > On Tuesday, June 10, 2014 12:53:41 PM UTC+5:30, Mark Walkom wrote: > > > How much RAM per node, what java flavour and version, what ES version? > > > > Are the logs showing any OOM? > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 10 June 2014 17:16, Gaurav Arora [gaurav...@gmail.com](mailto:gaurav...@gmail.com) wrote: > > > > > I'm having a very odd problem with one of my elasticsearch clusters. The > > > master node on the cluster crashes randomly. The cluster is running on 3 > > > different ec2 instances. > > > > > > My cluster configuration is: > > > 3 nodes (1 master, all data nodes) > > > 600 GB of data (3k IOPS EBS volumes) > > > 700 million documents > > > > > > There are no hprof files anywhere so I don't think this was a memory > > > problem. I have set the log level to DEBUG but there is no information in > > > them at all. > > > > > > I'm not sure where to start debugging this problem. I realise my post is > > > not very helpful but if I could get pointers on where to start I will > > > produce logs. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a3752511-ebfd-465b-b55b-3e2a28650350%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3752511-ebfd-465b-b55b-3e2a28650350%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Gaurav\_Arora](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gaurav_arora/32/1492_2.png) [@Gaurav\_Arora](https://discuss.elastic.co/u/Gaurav_Arora)\ **Post date:** [June 10, 2014, 9:59am UTC](https://discuss.elastic.co/t/master-node-keeps-crashing/18007/5 "2014-06-10T09:59:12Z")
The max file descriptors are all set to 64k. This is the output from one of the slave nodes - [http://pastebin.com/RdmZsJbH](http://pastebin.com/RdmZsJbH) On Tue, Jun 10, 2014 at 2:50 PM, sirkubax [jakubxmuszynski@googlemail.com](mailto:jakubxmuszynski@googlemail.com) wrote: > curl -XGET '[http://localhost:9200/\_nodes/\_all/process?pretty=true](http://localhost:9200/_nodes/_all/process?pretty=true)' |less > > Have you been considering "max\_file\_descriptors"? > > W dniu wtorek, 10 czerwca 2014 09:36:35 UTC+2 użytkownik Gaurav Arora > napisał: > > > I am using the latest openjdk version 7 installed from ubuntu repos. > > > > ubuntu@es1:~$ java -version > > java version "1.7.0\_51" > > OpenJDK Runtime Environment (IcedTea 2.4.6) (7u51-2.4.6-1ubuntu4) > > OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode) > > > > ES is set to run with -Xms14075m -Xmx14075m with bootstrap.mlockall set > > to true. > > > > The instance is a r3.large instance. > > > > There is no OOM in the logs anywhere, the last line of the log shows that > > an index was created about 15-20 minutes before the crash. Thats it. > > > > On Tuesday, June 10, 2014 12:53:41 PM UTC+5:30, Mark Walkom wrote: > > > > > How much RAM per node, what java flavour and version, what ES version? > > > > > > Are the logs showing any OOM? > > > > > > Regards, > > > Mark Walkom > > > > > > Infrastructure Engineer > > > Campaign Monitor > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > > > On 10 June 2014 17:16, Gaurav Arora [gaurav...@gmail.com](mailto:gaurav...@gmail.com) wrote: > > > > > > > I'm having a very odd problem with one of my elasticsearch clusters. > > > > The master node on the cluster crashes randomly. The cluster is running on > > > > 3 different ec2 instances. > > > > > > > > My cluster configuration is: > > > > 3 nodes (1 master, all data nodes) > > > > 600 GB of data (3k IOPS EBS volumes) > > > > 700 million documents > > > > > > > > There are no hprof files anywhere so I don't think this was a memory > > > > problem. I have set the log level to DEBUG but there is no information in > > > > them at all. > > > > > > > > I'm not sure where to start debugging this problem. I realise my post > > > > is not very helpful but if I could get pointers on where to start I will > > > > produce logs. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/36f06a1c-1c5d-4468-87c8-920f56b30efd%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/J4\_sNWls2dA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/J4_sNWls2dA/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/a3752511-ebfd-465b-b55b-3e2a28650350%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a3752511-ebfd-465b-b55b-3e2a28650350%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/a3752511-ebfd-465b-b55b-3e2a28650350%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a3752511-ebfd-465b-b55b-3e2a28650350%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJQ9OU1LPPVeEWCQ8gGgnXQ\_QkbdqUmMkEG70UqFqOx\_MtXvvA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJQ9OU1LPPVeEWCQ8gGgnXQ_QkbdqUmMkEG70UqFqOx_MtXvvA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:23am UTC](https://discuss.elastic.co/t/master-node-keeps-crashing/18007/6 "2017-07-06T01:23:37Z")