I'm running Elasticsearch 1.0 on an Ubuntu 13.10, laptop, strictly locally, as part of a development platform. Through my neglect of administrative hygiene, my Elasticsearch installation is in a state where when I start it up it's sometimes slow and sometimes completely unresponsive. Sometimes it works just fine on start up.
Specifically, my previous practice was to boot up the laptop, open a terminal window, and invoke
/usr/share/elasticsearch/bin/elasticsearch
in the foreground, and then ignore the following warnings:
log4j:WARN No appenders could be found for logger (node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Before shutting down the laptop, I would issue ctrl+c to that terminal window.
More recently, I my practice has been to invoke
curl -XPOST 'localhost:9200/_shutdown'
before starting Elasticsearch, and then to start Elasticsearch in the background. Before shutting down the laptop, I again invoke
curl -XPOST 'localhost:9200/_shutdown'
Here are some examples of what I've seen when Elasticsearch is unresponsive
ps x | grep elasticsearch
3931 pts/1 Sl 0:13 /usr/lib/jvm/default-java/bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.0.0.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* org.elasticsearch.bootstrap.Elasticsearch
{"error":"IndexFailedEngineException[[testindex5][2] Index failed for [chart#25]]; nested: OutOfMemoryError[Java heap space]; ","status":500}
Related to Marvel, on issuing ctrl+c in a terminal window running bin/elasticsearch in the foreground, I've seen
Exception in thread "Thread-1" java.lang.NullPointerException
at org.elasticsearch.marvel.agent.exporter.ESExporter.doStop(ESExporter.java:269)
at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105)
at org.elasticsearch.marvel.agent.AgentService.doStop(AgentService.java:180)
at org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105)
at org.elasticsearch.node.internal.InternalNode.stop(InternalNode.java:286)
at org.elasticsearch.node.internal.InternalNode.close(InternalNode.java:296)
at org.elasticsearch.bootstrap.Bootstrap$1.run(Bootstrap.java:73)
Also, immediately after starting Elasticsearch and opening the Marvel overview window, at different times I've seen in the Marvel overview window
Oops! FacetPhaseExecutionException[Facet [0]: (value) field [primaries.indexing.index_total] not found]
and
Oops! SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed]
One guess is that I've been blithely generating one logstash index per development day. If I have, I've certainly been neglecting these indexes. Otherwise, I've had no more than three test indexes in existence at one time, none with more than 25000 documents.
What can I do now to restore my Elasticsearch installation to robust health? Then what should I do to keep my
Elasticsearch installation healthy?