# Repairing effects of careless Elasticsearch administration **URL:** \ **Category:** Elasticsearch\ **Created:** [October 7, 2014, 3:02pm UTC](https://discuss.elastic.co/t/repairing-effects-of-careless-elasticsearch-administration/20117 "2014-10-07T15:02:30Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![pitaga1](https://avatars.discourse-cdn.com/v4/letter/p/85e7bf/32.png) [@pitaga1](https://discuss.elastic.co/u/pitaga1)\ **Post date:** [October 7, 2014, 3:02pm UTC](https://discuss.elastic.co/t/repairing-effects-of-careless-elasticsearch-administration/20117/1 "2014-10-07T15:02:30Z")
I'm running Elasticsearch 1.0 on an Ubuntu 13.10, laptop, strictly locally, as part of a development platform. Through my neglect of administrative hygiene, my Elasticsearch installation is in a state where when I start it up it's sometimes slow and sometimes completely unresponsive. Sometimes it works just fine on start up. Specifically, my previous practice was to boot up the laptop, open a terminal window, and invoke ``` /usr/share/elasticsearch/bin/elasticsearch ``` in the foreground, and then ignore the following warnings: ``` log4j:WARN No appenders could be found for logger (node). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for ``` more info. Before shutting down the laptop, I would issue ctrl+c to that terminal window. More recently, I my practice has been to invoke ``` curl -XPOST 'localhost:9200/_shutdown' ``` before starting Elasticsearch, and then to start Elasticsearch in the background. Before shutting down the laptop, I again invoke ``` curl -XPOST 'localhost:9200/_shutdown' ``` Here are some examples of what I've seen when Elasticsearch is unresponsive ``` ps x | grep elasticsearch 3931 pts/1 Sl 0:13 /usr/lib/jvm/default-java/bin/java -Xms256m ``` -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.0.0.jar:/usr/share/elasticsearch/lib/_:/usr/share/elasticsearch/lib/sigar/_ org.elasticsearch.bootstrap.Elasticsearch ``` {"error":"IndexFailedEngineException[[testindex5][2] Index failed for ``` [chart#25]]; nested: OutOfMemoryError[Java heap space]; ","status":500} Related to Marvel, on issuing ctrl+c in a terminal window running bin/elasticsearch in the foreground, I've seen ``` Exception in thread "Thread-1" java.lang.NullPointerException at ``` org.elasticsearch.marvel.agent.exporter.ESExporter.doStop(ESExporter.java:269) ``` at ``` org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105) ``` at ``` org.elasticsearch.marvel.agent.AgentService.doStop(AgentService.java:180) at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105) ``` at ``` org.elasticsearch.node.internal.InternalNode.stop(InternalNode.java:286) at org.elasticsearch.node.internal.InternalNode.close(InternalNode.java:296) at org.elasticsearch.bootstrap.Bootstrap$1.run(Bootstrap.java:73) Also, immediately after starting Elasticsearch and opening the Marvel overview window, at different times I've seen in the Marvel overview window ``` Oops! FacetPhaseExecutionException[Facet [0]: (value) field ``` [primaries.indexing.index\_total] not found] and ``` Oops! SearchPhaseExecutionException[Failed to execute phase ``` [query\_fetch], all shards failed] One guess is that I've been blithely generating one logstash index per development day. If I have, I've certainly been neglecting these indexes. Otherwise, I've had no more than three test indexes in existence at one time, none with more than 25000 documents. What can I do now to restore my Elasticsearch installation to robust health? Then what should I do to keep my Elasticsearch installation healthy? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 7, 2014, 9:15pm UTC](https://discuss.elastic.co/t/repairing-effects-of-careless-elasticsearch-administration/20117/2 "2014-10-07T21:15:52Z")
If you are on Ubuntu you should be using the .deb package and the service script to stop and start things. OOM means you have too much data for ES to handle, which is no surprise if only you have 1G of heap. How much data are you storing, how many indexes? Marvel should tell you this, but if you can't open it then install something like ElasticHQ or kopf. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 8 October 2014 02:02, Pitaga [achats@blarg.net](mailto:achats@blarg.net) wrote: > I'm running Elasticsearch 1.0 on an Ubuntu 13.10, laptop, strictly > locally, as part of a development platform. Through my neglect of > administrative hygiene, my Elasticsearch installation is in a state where > when I start it up it's sometimes slow and sometimes completely > unresponsive. Sometimes it works just fine on start up. > > Specifically, my previous practice was to boot up the laptop, open a > terminal window, and invoke > > ``` > /usr/share/elasticsearch/bin/elasticsearch > > ``` > > in the foreground, and then ignore the following warnings: > > ``` > log4j:WARN No appenders could be found for logger (node). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for > > ``` > > more info. > > Before shutting down the laptop, I would issue ctrl+c to that terminal > window. > > More recently, I my practice has been to invoke > > ``` > curl -XPOST 'localhost:9200/_shutdown' > > ``` > > before starting Elasticsearch, and then to start Elasticsearch in the > background. Before shutting down the laptop, I again invoke > > ``` > curl -XPOST 'localhost:9200/_shutdown' > > ``` > > Here are some examples of what I've seen when Elasticsearch is > unresponsive > > ``` > ps x | grep elasticsearch > 3931 pts/1 Sl 0:13 /usr/lib/jvm/default-java/bin/java -Xms256m > > ``` > > -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError > -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch > -cp > :/usr/share/elasticsearch/lib/elasticsearch-1.0.0.jar:/usr/share/elasticsearch/lib/_:/usr/share/elasticsearch/lib/sigar/_ > org.elasticsearch.bootstrap.Elasticsearch > > ``` > {"error":"IndexFailedEngineException[[testindex5][2] Index failed for > > ``` > > [chart#25]]; nested: OutOfMemoryError[Java heap space]; ","status":500} > > Related to Marvel, on issuing ctrl+c in a terminal window running > bin/elasticsearch in the foreground, I've seen > > ``` > Exception in thread "Thread-1" java.lang.NullPointerException > at > > ``` > > org.elasticsearch.marvel.agent.exporter.ESExporter.doStop(ESExporter.java:269) > > ``` > at > > ``` > > org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105) > > ``` > at > > ``` > > org.elasticsearch.marvel.agent.AgentService.doStop(AgentService.java:180) > at > org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105) > > ``` > at > > ``` > > org.elasticsearch.node.internal.InternalNode.stop(InternalNode.java:286) > at > org.elasticsearch.node.internal.InternalNode.close(InternalNode.java:296) > at org.elasticsearch.bootstrap.Bootstrap$1.run(Bootstrap.java:73) > > Also, immediately after starting Elasticsearch and opening the Marvel > overview window, at different times I've seen in the Marvel overview window > > ``` > Oops! FacetPhaseExecutionException[Facet [0]: (value) field > > ``` > > [primaries.indexing.index\_total] not found] > > and > > ``` > Oops! SearchPhaseExecutionException[Failed to execute phase > > ``` > > [query\_fetch], all shards failed] > > One guess is that I've been blithely generating one logstash index per > development day. If I have, I've certainly been neglecting these indexes. > Otherwise, I've had no more than three test indexes in existence at one > time, none with more than 25000 documents. > > What can I do now to restore my Elasticsearch installation to robust > health? Then what should I do to keep my > Elasticsearch installation healthy? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bBfH19Gap%2BO\_Fdx1pOvrULFE6x%2BEYTWBFC\_W4LBBZm%2Bw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bBfH19Gap%2BO_Fdx1pOvrULFE6x%2BEYTWBFC_W4LBBZm%2Bw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![pitaga1](https://avatars.discourse-cdn.com/v4/letter/p/85e7bf/32.png) [@pitaga1](https://discuss.elastic.co/u/pitaga1)\ **Post date:** [October 8, 2014, 7:49pm UTC](https://discuss.elastic.co/t/repairing-effects-of-careless-elasticsearch-administration/20117/3 "2014-10-08T19:49:22Z")
Thanks, Mark. This seems to solve my problems. I followed the instructions in > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. to set up use of the service script. While I was at it I increased heap to 4G. So far, so good. On Tuesday, October 7, 2014 2:16:19 PM UTC-7, Mark Walkom wrote: > If you are on Ubuntu you should be using the .deb package and the service > script to stop and start things. > OOM means you have too much data for ES to handle, which is no surprise if > only you have 1G of heap. > > How much data are you storing, how many indexes? Marvel should tell you > this, but if you can't open it then install something like ElasticHQ or > kopf. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 8 October 2014 02:02, Pitaga \<[ach...@blarg.net](mailto:ach...@blarg.net) \\> wrote: > > > I'm running Elasticsearch 1.0 on an Ubuntu 13.10, laptop, strictly > > locally, as part of a development platform. Through my neglect of > > administrative hygiene, my Elasticsearch installation is in a state where > > when I start it up it's sometimes slow and sometimes completely > > unresponsive. Sometimes it works just fine on start up. > > > > Specifically, my previous practice was to boot up the laptop, open a > > terminal window, and invoke > > > > ``` > > /usr/share/elasticsearch/bin/elasticsearch > > > > ``` > > > > in the foreground, and then ignore the following warnings: > > > > ``` > > log4j:WARN No appenders could be found for logger (node). > > log4j:WARN Please initialize the log4j system properly. > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for > > > > ``` > > > > more info. > > > > Before shutting down the laptop, I would issue ctrl+c to that terminal > > window. > > > > More recently, I my practice has been to invoke > > > > ``` > > curl -XPOST 'localhost:9200/_shutdown' > > > > ``` > > > > before starting Elasticsearch, and then to start Elasticsearch in the > > background. Before shutting down the laptop, I again invoke > > > > ``` > > curl -XPOST 'localhost:9200/_shutdown' > > > > ``` > > > > Here are some examples of what I've seen when Elasticsearch is > > unresponsive > > > > ``` > > ps x | grep elasticsearch > > 3931 pts/1 Sl 0:13 /usr/lib/jvm/default-java/bin/java -Xms256m > > > > ``` > > > > -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC > > -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 > > -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError > > -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch > > -cp > > :/usr/share/elasticsearch/lib/elasticsearch-1.0.0.jar:/usr/share/elasticsearch/lib/_:/usr/share/elasticsearch/lib/sigar/_ > > org.elasticsearch.bootstrap.Elasticsearch > > > > ``` > > {"error":"IndexFailedEngineException[[testindex5][2] Index failed for > > > > ``` > > > > [chart#25]]; nested: OutOfMemoryError[Java heap space]; ","status":500} > > > > Related to Marvel, on issuing ctrl+c in a terminal window running > > bin/elasticsearch in the foreground, I've seen > > > > ``` > > Exception in thread "Thread-1" java.lang.NullPointerException > > at > > > > ``` > > > > org.elasticsearch.marvel.agent.exporter.ESExporter.doStop(ESExporter.java:269) > > > > ``` > > at > > > > ``` > > > > org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105) > > > > ``` > > at > > > > ``` > > > > org.elasticsearch.marvel.agent.AgentService.doStop(AgentService.java:180) > > at > > org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105) > > > > ``` > > at > > > > ``` > > > > org.elasticsearch.node.internal.InternalNode.stop(InternalNode.java:286) > > at > > org.elasticsearch.node.internal.InternalNode.close(InternalNode.java:296) > > at org.elasticsearch.bootstrap.Bootstrap$1.run(Bootstrap.java:73) > > > > Also, immediately after starting Elasticsearch and opening the Marvel > > overview window, at different times I've seen in the Marvel overview window > > > > ``` > > Oops! FacetPhaseExecutionException[Facet [0]: (value) field > > > > ``` > > > > [primaries.indexing.index\_total] not found] > > > > and > > > > ``` > > Oops! SearchPhaseExecutionException[Failed to execute phase > > > > ``` > > > > [query\_fetch], all shards failed] > > > > One guess is that I've been blithely generating one logstash index per > > development day. If I have, I've certainly been neglecting these indexes. > > Otherwise, I've had no more than three test indexes in existence at one > > time, none with more than 25000 documents. > > > > What can I do now to restore my Elasticsearch installation to robust > > health? Then what should I do to keep my > > Elasticsearch installation healthy? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ec06385a-e113-494d-a167-efcf63e75a81%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ec06385a-e113-494d-a167-efcf63e75a81%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:57am UTC](https://discuss.elastic.co/t/repairing-effects-of-careless-elasticsearch-administration/20117/4 "2017-07-06T00:57:26Z")