# Strange problem: my ES server almost lost all its data. (All shards failed) **URL:** \ **Category:** Elasticsearch\ **Created:** [May 1, 2014, 9:46pm UTC](https://discuss.elastic.co/t/strange-problem-my-es-server-almost-lost-all-its-data-all-shards-failed/17308 "2014-05-01T21:46:33Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Patrick\_Proniewski](https://avatars.discourse-cdn.com/v4/letter/p/96bed5/32.png) [@Patrick\_Proniewski](https://discuss.elastic.co/u/Patrick_Proniewski)\ **Post date:** [May 1, 2014, 9:46pm UTC](https://discuss.elastic.co/t/strange-problem-my-es-server-almost-lost-all-its-data-all-shards-failed/17308/1 "2014-05-01T21:46:33Z")
Hello, I'm running a small server with logstash, ES, Kibana. Tonight, I've restarted my ES process. Very bad idea: it restarted with lots of errors, and finally "lost" all its data. Basically, before restart, I've had: elasticsearch/nodes/0/indices/logstash-2014.\* elasticsearch/nodes/0/\_state/ after restart, I've had: elasticsearch/nodes/0/indices/logstash-2014.\* elasticsearch/nodes/0/\_state/ elasticsearch/nodes/1/indices/logstash-2014.05.01 elasticsearch/nodes/1/\_state/ Then, Kibana was not able to find anything (dashboards lost, etc.). I've stopped Logstash, stopped Elasticsearch, waited a bit and checked everything is down, then restarted ES. It looked OK, then I've restarted Logstash, and I was able to access my dashboards again. I've just lost 15 minutes of data. Now I can see that elasticsearch/nodes/0 is the current working directory, and I can browse old data and current data. elasticsearch/nodes/1 is not used anymore. I'm running FreeBSD, and used the service command to restart ES. When attempting the second shutdown, the script wouldn't find the pid file, so I've had to kill the Java process. I don't understand what happened. But I don't feel comfortable putting ES in production. Full log for first and second restart here: [http://patpro.net/elastic.log](http://patpro.net/elastic.log) Any idea? Regards, Patrick -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/DEC08780-FC7C-44F7-B7B8-B70215060351%40patpro.net](https://groups.google.com/d/msgid/elasticsearch/DEC08780-FC7C-44F7-B7B8-B70215060351%40patpro.net). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [May 2, 2014, 7:38am UTC](https://discuss.elastic.co/t/strange-problem-my-es-server-almost-lost-all-its-data-all-shards-failed/17308/2 "2014-05-02T07:38:32Z")
On FreeBSD, do you have multicast on IPv6 enabled? You should disable IPv6 on the JVM. Seems you received a severe network error from the OS. Jörg On Thu, May 1, 2014 at 11:46 PM, Patrick Proniewski \< [elasticsearch@patpro.net](mailto:elasticsearch@patpro.net)\> wrote: > Hello, > > I'm running a small server with logstash, ES, Kibana. Tonight, I've > restarted my ES process. Very bad idea: it restarted with lots of errors, > and finally "lost" all its data. > Basically, before restart, I've had: > > elasticsearch/nodes/0/indices/logstash-2014.\* > elasticsearch/nodes/0/\_state/ > > after restart, I've had: > > elasticsearch/nodes/0/indices/logstash-2014.\* > elasticsearch/nodes/0/\_state/ > elasticsearch/nodes/1/indices/logstash-2014.05.01 > elasticsearch/nodes/1/\_state/ > > Then, Kibana was not able to find anything (dashboards lost, etc.). > > I've stopped Logstash, stopped Elasticsearch, waited a bit and checked > everything is down, then restarted ES. It looked OK, then I've restarted > Logstash, and I was able to access my dashboards again. I've just lost 15 > minutes of data. > Now I can see that elasticsearch/nodes/0 is the current working directory, > and I can browse old data and current data. > elasticsearch/nodes/1 is not used anymore. > > I'm running FreeBSD, and used the service command to restart ES. When > attempting the second shutdown, the script wouldn't find the pid file, so > I've had to kill the Java process. > > I don't understand what happened. But I don't feel comfortable putting ES > in production. Full log for first and second restart here: \< > [http://patpro.net/elastic.log](http://patpro.net/elastic.log)\> > > Any idea? > Regards, > Patrick > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/DEC08780-FC7C-44F7-B7B8-B70215060351%40patpro.net](https://groups.google.com/d/msgid/elasticsearch/DEC08780-FC7C-44F7-B7B8-B70215060351%40patpro.net) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG2Dvb2RTGdyukXOKS1DYGsnDTNQLnzCX%2Ba%2Bx%2B-KiuXjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG2Dvb2RTGdyukXOKS1DYGsnDTNQLnzCX%2Ba%2Bx%2B-KiuXjQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Patrick\_Proniewski](https://avatars.discourse-cdn.com/v4/letter/p/96bed5/32.png) [@Patrick\_Proniewski](https://discuss.elastic.co/u/Patrick_Proniewski)\ **Post date:** [May 2, 2014, 8:23am UTC](https://discuss.elastic.co/t/strange-problem-my-es-server-almost-lost-all-its-data-all-shards-failed/17308/3 "2014-05-02T08:23:10Z")
Hi Jörg, Thank you for your reply. The service script includes an option that might deal with IPv6, but it's not active: # Force the JVM to use IPv4 stack # elasticshearch\_props"-Djava.net.preferIPv4Stack=true" ([http://svnweb.freebsd.org/ports/head/textproc/elasticsearch/files/elasticsearch.in?revision=349955](http://svnweb.freebsd.org/ports/head/textproc/elasticsearch/files/elasticsearch.in?revision=349955)) In past years, I used to disable IPv6 everywhere (kernel, ports compilation, etc.) but now I don't bother anymore. Do you mean I should use this option to force IPv4? Thanks, Patrick On 2 mai 2014, at 09:38, [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > On FreeBSD, do you have multicast on IPv6 enabled? You should disable IPv6 > on the JVM. > > Seems you received a severe network error from the OS. > > Jörg > > On Thu, May 1, 2014 at 11:46 PM, Patrick Proniewski \< > [elasticsearch@patpro.net](mailto:elasticsearch@patpro.net)\> wrote: > > > Hello, > > > > I'm running a small server with logstash, ES, Kibana. Tonight, I've > > restarted my ES process. Very bad idea: it restarted with lots of errors, > > and finally "lost" all its data. > > Basically, before restart, I've had: > > > > elasticsearch/nodes/0/indices/logstash-2014.\* > > elasticsearch/nodes/0/\_state/ > > > > after restart, I've had: > > > > elasticsearch/nodes/0/indices/logstash-2014.\* > > elasticsearch/nodes/0/\_state/ > > elasticsearch/nodes/1/indices/logstash-2014.05.01 > > elasticsearch/nodes/1/\_state/ > > > > Then, Kibana was not able to find anything (dashboards lost, etc.). > > > > I've stopped Logstash, stopped Elasticsearch, waited a bit and checked > > everything is down, then restarted ES. It looked OK, then I've restarted > > Logstash, and I was able to access my dashboards again. I've just lost 15 > > minutes of data. > > Now I can see that elasticsearch/nodes/0 is the current working directory, > > and I can browse old data and current data. > > elasticsearch/nodes/1 is not used anymore. > > > > I'm running FreeBSD, and used the service command to restart ES. When > > attempting the second shutdown, the script wouldn't find the pid file, so > > I've had to kill the Java process. > > > > I don't understand what happened. But I don't feel comfortable putting ES > > in production. Full log for first and second restart here: \< > > [http://patpro.net/elastic.log](http://patpro.net/elastic.log)\> > > > > Any idea? > > Regards, > > Patrick -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/AA0BC7BA-8856-4A23-A172-0601BC0B4FEE%40patpro.net](https://groups.google.com/d/msgid/elasticsearch/AA0BC7BA-8856-4A23-A172-0601BC0B4FEE%40patpro.net). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [May 2, 2014, 12:07pm UTC](https://discuss.elastic.co/t/strange-problem-my-es-server-almost-lost-all-its-data-all-shards-failed/17308/4 "2014-05-02T12:07:46Z")
Yes, you should use this option. Some FreeBSD kernels seem to have difficulties to run UDP multicast on IPv6 together with IPv4 properly, so I would like to suggest disabling IPv6 use on the JVM. Jörg On Fri, May 2, 2014 at 10:23 AM, Patrick Proniewski \< [elasticsearch@patpro.net](mailto:elasticsearch@patpro.net)\> wrote: > Hi Jörg, > > Thank you for your reply. > The service script includes an option that might deal with IPv6, but it's > not active: > > # Force the JVM to use IPv4 stack > > # elasticshearch\_props"-Djava.net.preferIPv4Stack=true" > > (\< > [http://svnweb.freebsd.org/ports/head/textproc/elasticsearch/files/elasticsearch.in?revision=349955](http://svnweb.freebsd.org/ports/head/textproc/elasticsearch/files/elasticsearch.in?revision=349955) > > > ) > > In past years, I used to disable IPv6 everywhere (kernel, ports > compilation, etc.) but now I don't bother anymore. > Do you mean I should use this option to force IPv4? > > Thanks, > Patrick > > On 2 mai 2014, at 09:38, [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > > > On FreeBSD, do you have multicast on IPv6 enabled? You should disable > > IPv6 > > on the JVM. > > > > Seems you received a severe network error from the OS. > > > > Jörg > > > > On Thu, May 1, 2014 at 11:46 PM, Patrick Proniewski \< > > [elasticsearch@patpro.net](mailto:elasticsearch@patpro.net)\> wrote: > > > > > Hello, > > > > > > I'm running a small server with logstash, ES, Kibana. Tonight, I've > > > restarted my ES process. Very bad idea: it restarted with lots of > > > errors, > > > and finally "lost" all its data. > > > Basically, before restart, I've had: > > > > > > elasticsearch/nodes/0/indices/logstash-2014.\* > > > elasticsearch/nodes/0/\_state/ > > > > > > after restart, I've had: > > > > > > elasticsearch/nodes/0/indices/logstash-2014.\* > > > elasticsearch/nodes/0/\_state/ > > > elasticsearch/nodes/1/indices/logstash-2014.05.01 > > > elasticsearch/nodes/1/\_state/ > > > > > > Then, Kibana was not able to find anything (dashboards lost, etc.). > > > > > > I've stopped Logstash, stopped Elasticsearch, waited a bit and checked > > > everything is down, then restarted ES. It looked OK, then I've restarted > > > Logstash, and I was able to access my dashboards again. I've just lost > > > 15 > > > minutes of data. > > > Now I can see that elasticsearch/nodes/0 is the current working > > > directory, > > > and I can browse old data and current data. > > > elasticsearch/nodes/1 is not used anymore. > > > > > > I'm running FreeBSD, and used the service command to restart ES. When > > > attempting the second shutdown, the script wouldn't find the pid file, > > > so > > > I've had to kill the Java process. > > > > > > I don't understand what happened. But I don't feel comfortable putting > > > ES > > > in production. Full log for first and second restart here: \< > > > [http://patpro.net/elastic.log](http://patpro.net/elastic.log)\> > > > > > > Any idea? > > > Regards, > > > Patrick > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/AA0BC7BA-8856-4A23-A172-0601BC0B4FEE%40patpro.net](https://groups.google.com/d/msgid/elasticsearch/AA0BC7BA-8856-4A23-A172-0601BC0B4FEE%40patpro.net) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGN\_zdUi9cCAoAgprBh-Lxtu\_g1ejSQWT\_nZU0fd\_YRTA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGN_zdUi9cCAoAgprBh-Lxtu_g1ejSQWT_nZU0fd_YRTA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Patrick\_Proniewski](https://avatars.discourse-cdn.com/v4/letter/p/96bed5/32.png) [@Patrick\_Proniewski](https://discuss.elastic.co/u/Patrick_Proniewski)\ **Post date:** [May 2, 2014, 1:00pm UTC](https://discuss.elastic.co/t/strange-problem-my-es-server-almost-lost-all-its-data-all-shards-failed/17308/5 "2014-05-02T13:00:02Z")
Thank you for the tip, Jörg. I've activated this option and carefully restarted. I've re-read yesterday's log file, and now I think may be the new ES instance started before the former one was completely terminated. This too can cause some network/socket trouble. I might try and add a short sleep into the restart command. On 2 mai 2014, at 14:07, [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > Yes, you should use this option. > > Some FreeBSD kernels seem to have difficulties to run UDP multicast on IPv6 > together with IPv4 properly, so I would like to suggest disabling IPv6 use > on the JVM. > > Jörg > > On Fri, May 2, 2014 at 10:23 AM, Patrick Proniewski \< > [elasticsearch@patpro.net](mailto:elasticsearch@patpro.net)\> wrote: > > > Hi Jörg, > > > > Thank you for your reply. > > The service script includes an option that might deal with IPv6, but it's > > not active: > > > > # Force the JVM to use IPv4 stack > > > > # elasticshearch\_props"-Djava.net.preferIPv4Stack=true" > > > > (\< > > [http://svnweb.freebsd.org/ports/head/textproc/elasticsearch/files/elasticsearch.in?revision=349955](http://svnweb.freebsd.org/ports/head/textproc/elasticsearch/files/elasticsearch.in?revision=349955) > > > > > ) > > > > In past years, I used to disable IPv6 everywhere (kernel, ports > > compilation, etc.) but now I don't bother anymore. > > Do you mean I should use this option to force IPv4? > > > > Thanks, > > Patrick -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CE3C61C8-3EC1-49A0-A6DC-F38432CF123C%40patpro.net](https://groups.google.com/d/msgid/elasticsearch/CE3C61C8-3EC1-49A0-A6DC-F38432CF123C%40patpro.net). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:32am UTC](https://discuss.elastic.co/t/strange-problem-my-es-server-almost-lost-all-its-data-all-shards-failed/17308/6 "2017-07-06T01:32:06Z")