Repairing effects of careless Elasticsearch administration

I'm running Elasticsearch 1.0 on an Ubuntu 13.10, laptop, strictly locally,
as part of a development platform. Through my neglect of administrative
hygiene, my Elasticsearch installation is in a state where when I start it
up it's sometimes slow and sometimes completely unresponsive. Sometimes it
works just fine on start up.

Specifically, my previous practice was to boot up the laptop, open a
terminal window, and invoke

/usr/share/elasticsearch/bin/elasticsearch 

in the foreground, and then ignore the following warnings:

log4j:WARN No appenders could be found for logger (node). 
log4j:WARN Please initialize the log4j system properly. 
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 

more info.

Before shutting down the laptop, I would issue ctrl+c to that terminal
window.

More recently, I my practice has been to invoke

curl -XPOST 'localhost:9200/_shutdown' 

before starting Elasticsearch, and then to start Elasticsearch in the
background. Before shutting down the laptop, I again invoke

curl -XPOST 'localhost:9200/_shutdown' 

Here are some examples of what I've seen when Elasticsearch is unresponsive

ps x | grep elasticsearch 
3931 pts/1    Sl     0:13 /usr/lib/jvm/default-java/bin/java -Xms256m 

-Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError
-Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch
-cp
:/usr/share/elasticsearch/lib/elasticsearch-1.0.0.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/
org.elasticsearch.bootstrap.Elasticsearch

{"error":"IndexFailedEngineException[[testindex5][2] Index failed for 

[chart#25]]; nested: OutOfMemoryError[Java heap space]; ","status":500}

Related to Marvel, on issuing ctrl+c in a terminal window running
bin/elasticsearch in the foreground, I've seen

Exception in thread "Thread-1" java.lang.NullPointerException 
    at 

org.elasticsearch.marvel.agent.exporter.ESExporter.doStop(ESExporter.java:269)

    at 

org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105)

    at 

org.elasticsearch.marvel.agent.AgentService.doStop(AgentService.java:180)
at
org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105)

    at 

org.elasticsearch.node.internal.InternalNode.stop(InternalNode.java:286)
at
org.elasticsearch.node.internal.InternalNode.close(InternalNode.java:296)
at org.elasticsearch.bootstrap.Bootstrap$1.run(Bootstrap.java:73)

Also, immediately after starting Elasticsearch and opening the Marvel
overview window, at different times I've seen in the Marvel overview window

Oops! FacetPhaseExecutionException[Facet [0]: (value) field 

[primaries.indexing.index_total] not found]

and

Oops! SearchPhaseExecutionException[Failed to execute phase 

[query_fetch], all shards failed]

One guess is that I've been blithely generating one logstash index per
development day. If I have, I've certainly been neglecting these indexes.
Otherwise, I've had no more than three test indexes in existence at one
time, none with more than 25000 documents.

What can I do now to restore my Elasticsearch installation to robust
health? Then what should I do to keep my
Elasticsearch installation healthy?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

If you are on Ubuntu you should be using the .deb package and the service
script to stop and start things.
OOM means you have too much data for ES to handle, which is no surprise if
only you have 1G of heap.

How much data are you storing, how many indexes? Marvel should tell you
this, but if you can't open it then install something like ElasticHQ or
kopf.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 8 October 2014 02:02, Pitaga achats@blarg.net wrote:

I'm running Elasticsearch 1.0 on an Ubuntu 13.10, laptop, strictly
locally, as part of a development platform. Through my neglect of
administrative hygiene, my Elasticsearch installation is in a state where
when I start it up it's sometimes slow and sometimes completely
unresponsive. Sometimes it works just fine on start up.

Specifically, my previous practice was to boot up the laptop, open a
terminal window, and invoke

/usr/share/elasticsearch/bin/elasticsearch

in the foreground, and then ignore the following warnings:

log4j:WARN No appenders could be found for logger (node).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for

more info.

Before shutting down the laptop, I would issue ctrl+c to that terminal
window.

More recently, I my practice has been to invoke

curl -XPOST 'localhost:9200/_shutdown'

before starting Elasticsearch, and then to start Elasticsearch in the
background. Before shutting down the laptop, I again invoke

curl -XPOST 'localhost:9200/_shutdown'

Here are some examples of what I've seen when Elasticsearch is
unresponsive

ps x | grep elasticsearch
3931 pts/1    Sl     0:13 /usr/lib/jvm/default-java/bin/java -Xms256m

-Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError
-Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch
-cp
:/usr/share/elasticsearch/lib/elasticsearch-1.0.0.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/
org.elasticsearch.bootstrap.Elasticsearch

{"error":"IndexFailedEngineException[[testindex5][2] Index failed for

[chart#25]]; nested: OutOfMemoryError[Java heap space]; ","status":500}

Related to Marvel, on issuing ctrl+c in a terminal window running
bin/elasticsearch in the foreground, I've seen

Exception in thread "Thread-1" java.lang.NullPointerException
    at

org.elasticsearch.marvel.agent.exporter.ESExporter.doStop(ESExporter.java:269)

    at

org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105)

    at

org.elasticsearch.marvel.agent.AgentService.doStop(AgentService.java:180)
at
org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105)

    at

org.elasticsearch.node.internal.InternalNode.stop(InternalNode.java:286)
at
org.elasticsearch.node.internal.InternalNode.close(InternalNode.java:296)
at org.elasticsearch.bootstrap.Bootstrap$1.run(Bootstrap.java:73)

Also, immediately after starting Elasticsearch and opening the Marvel
overview window, at different times I've seen in the Marvel overview window

Oops! FacetPhaseExecutionException[Facet [0]: (value) field

[primaries.indexing.index_total] not found]

and

Oops! SearchPhaseExecutionException[Failed to execute phase

[query_fetch], all shards failed]

One guess is that I've been blithely generating one logstash index per
development day. If I have, I've certainly been neglecting these indexes.
Otherwise, I've had no more than three test indexes in existence at one
time, none with more than 25000 documents.

What can I do now to restore my Elasticsearch installation to robust
health? Then what should I do to keep my
Elasticsearch installation healthy?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bBfH19Gap%2BO_Fdx1pOvrULFE6x%2BEYTWBFC_W4LBBZm%2Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Thanks, Mark. This seems to solve my problems. I followed the instructions
in

www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-service.html
to set up use of the service script. While I was at it I increased heap to
4G. So far, so good.

On Tuesday, October 7, 2014 2:16:19 PM UTC-7, Mark Walkom wrote:

If you are on Ubuntu you should be using the .deb package and the service
script to stop and start things.
OOM means you have too much data for ES to handle, which is no surprise if
only you have 1G of heap.

How much data are you storing, how many indexes? Marvel should tell you
this, but if you can't open it then install something like ElasticHQ or
kopf.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 8 October 2014 02:02, Pitaga <ach...@blarg.net <javascript:>> wrote:

I'm running Elasticsearch 1.0 on an Ubuntu 13.10, laptop, strictly
locally, as part of a development platform. Through my neglect of
administrative hygiene, my Elasticsearch installation is in a state where
when I start it up it's sometimes slow and sometimes completely
unresponsive. Sometimes it works just fine on start up.

Specifically, my previous practice was to boot up the laptop, open a
terminal window, and invoke

/usr/share/elasticsearch/bin/elasticsearch 

in the foreground, and then ignore the following warnings:

log4j:WARN No appenders could be found for logger (node). 
log4j:WARN Please initialize the log4j system properly. 
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 

more info.

Before shutting down the laptop, I would issue ctrl+c to that terminal
window.

More recently, I my practice has been to invoke

curl -XPOST 'localhost:9200/_shutdown' 

before starting Elasticsearch, and then to start Elasticsearch in the
background. Before shutting down the laptop, I again invoke

curl -XPOST 'localhost:9200/_shutdown' 

Here are some examples of what I've seen when Elasticsearch is
unresponsive

ps x | grep elasticsearch 
3931 pts/1    Sl     0:13 /usr/lib/jvm/default-java/bin/java -Xms256m 

-Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError
-Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch
-cp
:/usr/share/elasticsearch/lib/elasticsearch-1.0.0.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/
org.elasticsearch.bootstrap.Elasticsearch

{"error":"IndexFailedEngineException[[testindex5][2] Index failed for 

[chart#25]]; nested: OutOfMemoryError[Java heap space]; ","status":500}

Related to Marvel, on issuing ctrl+c in a terminal window running
bin/elasticsearch in the foreground, I've seen

Exception in thread "Thread-1" java.lang.NullPointerException 
    at 

org.elasticsearch.marvel.agent.exporter.ESExporter.doStop(ESExporter.java:269)

    at 

org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105)

    at 

org.elasticsearch.marvel.agent.AgentService.doStop(AgentService.java:180)
at
org.elasticsearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:105)

    at 

org.elasticsearch.node.internal.InternalNode.stop(InternalNode.java:286)
at
org.elasticsearch.node.internal.InternalNode.close(InternalNode.java:296)
at org.elasticsearch.bootstrap.Bootstrap$1.run(Bootstrap.java:73)

Also, immediately after starting Elasticsearch and opening the Marvel
overview window, at different times I've seen in the Marvel overview window

Oops! FacetPhaseExecutionException[Facet [0]: (value) field 

[primaries.indexing.index_total] not found]

and

Oops! SearchPhaseExecutionException[Failed to execute phase 

[query_fetch], all shards failed]

One guess is that I've been blithely generating one logstash index per
development day. If I have, I've certainly been neglecting these indexes.
Otherwise, I've had no more than three test indexes in existence at one
time, none with more than 25000 documents.

What can I do now to restore my Elasticsearch installation to robust
health? Then what should I do to keep my
Elasticsearch installation healthy?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/17c763ae-3b8b-4aa8-8df6-e8c75750c209%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ec06385a-e113-494d-a167-efcf63e75a81%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.