# Elasticsearch index getting reset **URL:** \ **Category:** Elasticsearch\ **Created:** [November 13, 2013, 11:59pm UTC](https://discuss.elastic.co/t/elasticsearch-index-getting-reset/14392 "2013-11-13T23:59:47Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![aj1](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@aj1](https://discuss.elastic.co/u/aj1)\ **Post date:** [November 13, 2013, 11:59pm UTC](https://discuss.elastic.co/t/elasticsearch-index-getting-reset/14392/1 "2013-11-13T23:59:47Z")
Hi Guys, I have a single node elasticsearch instance ( 0.90 version) running on a single machine ( 8GB RAM, dual core CPU) having RHEL 5.6, java 1\_6\_00 After having indexed close to 2 million documents, it runs fine for a few hours and then restarts on its own, wiping out the index in the process. I now need to reindex all the documents again. Any ideas on why this happens? Maximum file descriptors is set to 32k and the number of open file descriptors at any time does not even come close. So it cant be that. Here are the modifications i made to the default elasticsearch.yml file : index.number\_of\_shards: 5 index.cache.field.type: soft index.fielddata.cache: soft index.cache.field.expire: 5m indices.fielddata.cache.size: 10% indices.fielddata.cache.expire : 5m index.store.type: mmapfs bootstrap.mlockall: true discovery.zen.ping.multicast.enabled: false action.disable\_delete\_all\_indices: true script.disable\_dynamic: true I use the elasticsearch service wrapper to start and stop the instance. In the elasticsearch.conf file, i have set the heap size to 2GB : set.default.ES\_HEAP\_SIZE=2048 How do i go about diagnosing the issue? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [November 14, 2013, 7:02am UTC](https://discuss.elastic.co/t/elasticsearch-index-getting-reset/14392/2 "2013-11-14T07:02:37Z")
Hi, did you consult ES log files? Also I would consider switching to Java7. Regards, Lukáš Dne 14.11.2013 0:59 "ajoy" [ajoy.sojan@quartzy.com](mailto:ajoy.sojan@quartzy.com) napsal(a): > Hi Guys, > > I have a single node elasticsearch instance ( 0.90 version) running on a > single machine ( 8GB RAM, dual core CPU) having RHEL 5.6, java 1\_6\_00 > > After having indexed close to 2 million documents, it runs fine for a few > hours and then restarts on its own, wiping out the index in the process. I > now need to reindex all the documents again. > > Any ideas on why this happens? Maximum file descriptors is set to 32k and > the number of open file descriptors at any time does not even come close. > So it cant be that. > > Here are the modifications i made to the default elasticsearch.yml file : > > index.number\_of\_shards: 5 > index.cache.field.type: soft > index.fielddata.cache: soft > index.cache.field.expire: 5m > indices.fielddata.cache.size: 10% > indices.fielddata.cache.expire : 5m > index.store.type: mmapfs > bootstrap.mlockall: true > discovery.zen.ping.multicast.enabled: false > action.disable\_delete\_all\_indices: true > script.disable\_dynamic: true > > I use the elasticsearch service wrapper to start and stop the instance. In > the elasticsearch.conf file, i have set the heap size to 2GB : > > set.default.ES\_HEAP\_SIZE=2048 > > How do i go about diagnosing the issue? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![aj1](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@aj1](https://discuss.elastic.co/u/aj1)\ **Post date:** [November 14, 2013, 5:08pm UTC](https://discuss.elastic.co/t/elasticsearch-index-getting-reset/14392/3 "2013-11-14T17:08:58Z")
ES Log files does not have anything informative. -- after it crapped out [2013-11-14 11:22:12,597][DEBUG][action.search.type] [WEB2] All shards failed for phase: [query] -- after i restarted [2013-11-14 11:28:39,670][INFO][cluster.metadata] [WEB2] [[items\_index]] remove\_mapping [item] [2013-11-14 11:29:06,102][DEBUG][action.index] [WEB2] Sending mapping updated to master: index [items\_index] type [item] [2013-11-14 11:29:06,107][INFO][cluster.metadata] [WEB2] [items\_index] update\_mapping [item] (dynamic) switching to java7 is the next option. But this kind of behavior is worrysome. Either it should fail and crash and give a dump, or it should stop responding. It shouldnt just wipe out the index and pretend like nothing happened. On Wednesday, November 13, 2013 11:02:37 PM UTC-8, Lukáš Vlček wrote: > Hi, > > did you consult ES log files? > Also I would consider switching to Java7. > > Regards, > Lukáš > Dne 14.11.2013 0:59 "ajoy" \<[ajoy....@quartzy.com](mailto:ajoy....@quartzy.com) \\> napsal(a): > > > Hi Guys, > > > > I have a single node elasticsearch instance ( 0.90 version) running on a > > single machine ( 8GB RAM, dual core CPU) having RHEL 5.6, java 1\_6\_00 > > > > After having indexed close to 2 million documents, it runs fine for a few > > hours and then restarts on its own, wiping out the index in the process. I > > now need to reindex all the documents again. > > > > Any ideas on why this happens? Maximum file descriptors is set to 32k and > > the number of open file descriptors at any time does not even come close. > > So it cant be that. > > > > Here are the modifications i made to the default elasticsearch.yml file : > > > > index.number\_of\_shards: 5 > > index.cache.field.type: soft > > index.fielddata.cache: soft > > index.cache.field.expire: 5m > > indices.fielddata.cache.size: 10% > > indices.fielddata.cache.expire : 5m > > index.store.type: mmapfs > > bootstrap.mlockall: true > > discovery.zen.ping.multicast.enabled: false > > action.disable\_delete\_all\_indices: true > > script.disable\_dynamic: true > > > > I use the elasticsearch service wrapper to start and stop the instance. > > In the elasticsearch.conf file, i have set the heap size to 2GB : > > > > set.default.ES\_HEAP\_SIZE=2048 > > > > How do i go about diagnosing the issue? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [November 15, 2013, 3:41pm UTC](https://discuss.elastic.co/t/elasticsearch-index-getting-reset/14392/4 "2013-11-15T15:41:53Z")
Are you sure the index was wiped out? Was it deleted or corrupted? Was the service restarted or was another one started at the same time? It might be the latter, which would cause another data sub-directory to be created. How many directories do you have under ${path.data}/${cluster.name}/nodes? Cheers, Ivan On Thu, Nov 14, 2013 at 9:08 AM, ajoy [ajoy.sojan@quartzy.com](mailto:ajoy.sojan@quartzy.com) wrote: > ES Log files does not have anything informative. > > -- after it crapped out > [2013-11-14 11:22:12,597][DEBUG][action.search.type] [WEB2] All > shards failed for phase: [query] > > -- after i restarted > [2013-11-14 11:28:39,670][INFO][cluster.metadata] [WEB2] > [[items\_index]] remove\_mapping [item] > [2013-11-14 11:29:06,102][DEBUG][action.index] [WEB2] Sending > mapping updated to master: index [items\_index] type [item] > [2013-11-14 11:29:06,107][INFO][cluster.metadata] [WEB2] > [items\_index] update\_mapping [item] (dynamic) > > switching to java7 is the next option. > > But this kind of behavior is worrysome. > Either it should fail and crash and give a dump, or it should stop > responding. > It shouldnt just wipe out the index and pretend like nothing happened. > > On Wednesday, November 13, 2013 11:02:37 PM UTC-8, Lukáš Vlček wrote: > > > Hi, > > > > did you consult ES log files? > > Also I would consider switching to Java7. > > > > Regards, > > Lukáš > > Dne 14.11.2013 0:59 "ajoy" [ajoy....@quartzy.com](mailto:ajoy....@quartzy.com) napsal(a): > > > > > Hi Guys, > > > > > > I have a single node elasticsearch instance ( 0.90 version) running on a > > > single machine ( 8GB RAM, dual core CPU) having RHEL 5.6, java 1\_6\_00 > > > > > > After having indexed close to 2 million documents, it runs fine for a > > > few hours and then restarts on its own, wiping out the index in the > > > process. I now need to reindex all the documents again. > > > > > > Any ideas on why this happens? Maximum file descriptors is set to 32k > > > and the number of open file descriptors at any time does not even come > > > close. So it cant be that. > > > > > > Here are the modifications i made to the default elasticsearch.yml file : > > > > > > index.number\_of\_shards: 5 > > > index.cache.field.type: soft > > > index.fielddata.cache: soft > > > index.cache.field.expire: 5m > > > indices.fielddata.cache.size: 10% > > > indices.fielddata.cache.expire : 5m > > > index.store.type: mmapfs > > > bootstrap.mlockall: true > > > discovery.zen.ping.multicast.enabled: false > > > action.disable\_delete\_all\_indices: true > > > script.disable\_dynamic: true > > > > > > I use the elasticsearch service wrapper to start and stop the instance. > > > In the elasticsearch.conf file, i have set the heap size to 2GB : > > > > > > set.default.ES\_HEAP\_SIZE=2048 > > > > > > How do i go about diagnosing the issue? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![aj1](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@aj1](https://discuss.elastic.co/u/aj1)\ **Post date:** [November 15, 2013, 8:21pm UTC](https://discuss.elastic.co/t/elasticsearch-index-getting-reset/14392/5 "2013-11-15T20:21:08Z")
I am not sure what is happening. 2 things are clearly different. 1. docs count come down from 2 million to less than a 100. 2. store size comes down to around 100k ( from 690 MB) I am running the service as a console application. So I am sure it was not stopped at any poiint. I have one folder under ${path.data}/${cluster.name}/nodes - '0/' I had added the following to the config file : ``` *action.disable_delete_all_indices: true* ``` - ``` script.disable_dynamic: true* ``` In the most recent occurence, I observed that the store size remains at 690 MB, but the docs count went down to 80. Almost as if the index is getting reset somehow. On Friday, November 15, 2013 7:41:53 AM UTC-8, Ivan Brusic wrote: > Are you sure the index was wiped out? Was it deleted or corrupted? Was the > service restarted or was another one started at the same time? It might be > the latter, which would cause another data sub-directory to be created. > How many directories do you have under ${path.data}/${cluster.name}/ > nodes? > > Cheers, > > Ivan > > On Thu, Nov 14, 2013 at 9:08 AM, ajoy \<[ajoy....@quartzy.com](mailto:ajoy....@quartzy.com) \\>wrote: > > > ES Log files does not have anything informative. > > > > -- after it crapped out > > [2013-11-14 11:22:12,597][DEBUG][action.search.type] [WEB2] All > > shards failed for phase: [query] > > > > -- after i restarted > > [2013-11-14 11:28:39,670][INFO][cluster.metadata] [WEB2] > > [[items\_index]] remove\_mapping [item] > > [2013-11-14 11:29:06,102][DEBUG][action.index] [WEB2] > > Sending mapping updated to master: index [items\_index] type [item] > > [2013-11-14 11:29:06,107][INFO][cluster.metadata] [WEB2] > > [items\_index] update\_mapping [item] (dynamic) > > > > switching to java7 is the next option. > > > > But this kind of behavior is worrysome. > > Either it should fail and crash and give a dump, or it should stop > > responding. > > It shouldnt just wipe out the index and pretend like nothing happened. > > > > On Wednesday, November 13, 2013 11:02:37 PM UTC-8, Lukáš Vlček wrote: > > > > > Hi, > > > > > > did you consult ES log files? > > > Also I would consider switching to Java7. > > > > > > Regards, > > > Lukáš > > > Dne 14.11.2013 0:59 "ajoy" [ajoy....@quartzy.com](mailto:ajoy....@quartzy.com) napsal(a): > > > > > > > Hi Guys, > > > > > > > > I have a single node elasticsearch instance ( 0.90 version) running on > > > > a single machine ( 8GB RAM, dual core CPU) having RHEL 5.6, java 1\_6\_00 > > > > > > > > After having indexed close to 2 million documents, it runs fine for a > > > > few hours and then restarts on its own, wiping out the index in the > > > > process. I now need to reindex all the documents again. > > > > > > > > Any ideas on why this happens? Maximum file descriptors is set to 32k > > > > and the number of open file descriptors at any time does not even come > > > > close. So it cant be that. > > > > > > > > Here are the modifications i made to the default elasticsearch.yml file > > > > : > > > > > > > > index.number\_of\_shards: 5 > > > > index.cache.field.type: soft > > > > index.fielddata.cache: soft > > > > index.cache.field.expire: 5m > > > > indices.fielddata.cache.size: 10% > > > > indices.fielddata.cache.expire : 5m > > > > index.store.type: mmapfs > > > > bootstrap.mlockall: true > > > > discovery.zen.ping.multicast.enabled: false > > > > action.disable\_delete\_all\_indices: true > > > > script.disable\_dynamic: true > > > > > > > > I use the elasticsearch service wrapper to start and stop the instance. > > > > In the elasticsearch.conf file, i have set the heap size to 2GB : > > > > > > > > set.default.ES\_HEAP\_SIZE=2048 > > > > > > > > How do i go about diagnosing the issue? > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:06am UTC](https://discuss.elastic.co/t/elasticsearch-index-getting-reset/14392/6 "2017-07-06T02:06:40Z")