Elasticsearch shuts down for no reason


(Eitan Vesely) #1

Hi Guys,
i've installed ES a month ago and its working just fine.

today, for some reason, ES just went down for no visible reason:

here is what i see in the log file :

[2014-08-23 16:47:11,272][DEBUG][action.search.type ] [Plunderer]
[g30nm0bi2j663tgu6ud][1], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad]
lastShard [true]
org.elasticsearch.search.SearchParseException: [g30nm0bi2j663tgu6ud][1]:
from[-1],size[-1]: Parse Failure [Failed to parse source
[{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"user_count","interval":"1h"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":""}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":1408811520255}}}]}}}}}}}},"size":0}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at
org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at
org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more
[2014-08-23 16:47:11,273][DEBUG][action.search.type ] [Plunderer]
[g30nm0bi2j663tgu6ud][0], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad]
org.elasticsearch.search.SearchParseException: [g30nm0bi2j663tgu6ud][0]:
from[-1],size[-1]: Parse Failure [Failed to parse source
[{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"user_count","interval":"1h"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"
"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":1408811520255}}}]}}}}}}}},"size":0}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at
org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at
org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more
[2014-08-23 16:49:05,401][INFO ][node ] [Plunderer]
stopping ...
[2014-08-23 16:49:06,737][INFO ][node ] [Plunderer]
stopped
[2014-08-23 16:49:06,737][INFO ][node ] [Plunderer]
closing ...
[2014-08-23 16:49:06,772][INFO ][node ] [Plunderer]
closed
[2014-08-23 20:02:09,693][WARN ][common.jna ] Unable to lock
JVM memory (ENOMEM). This can result in part of the JVM being swapped out.
Increase RLIMIT_MEMLOCK or run elasticsearch as root.
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet Abdol]
version[1.2.2], pid[2715], build[9902f08/2014-07-09T12:02:32Z]
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet Abdol]
initializing ...
[2014-08-23 20:02:09,940][INFO ][plugins ] [Ahmet Abdol]
loaded [], sites [kopf]
[2014-08-23 20:02:14,692][INFO ][node ] [Ahmet Abdol]
initialized

any ideas??

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

Something is stopping the service.

If you are on linux check the auth log, if anyone is using sudo to stop it
then you will see that logged. Otherwise, what else runs on the machine?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 24 August 2014 06:15, Eitan Vesely eitanjnk@gmail.com wrote:

Hi Guys,
i've installed ES a month ago and its working just fine.

today, for some reason, ES just went down for no visible reason:

here is what i see in the log file :

[2014-08-23 16:47:11,272][DEBUG][action.search.type ] [Plunderer]
[g30nm0bi2j663tgu6ud][1], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad]
lastShard [true]
org.elasticsearch.search.SearchParseException: [g30nm0bi2j663tgu6ud][1]:
from[-1],size[-1]: Parse Failure [Failed to parse source
[{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"user_count","interval":"1h"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":""}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":1408811520255}}}]}}}}}}}},"size":0}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at
org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at
org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more
[2014-08-23 16:47:11,273][DEBUG][action.search.type ] [Plunderer]
[g30nm0bi2j663tgu6ud][0], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad]
org.elasticsearch.search.SearchParseException: [g30nm0bi2j663tgu6ud][0]:
from[-1],size[-1]: Parse Failure [Failed to parse source
[{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"user_count","interval":"1h"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"
"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":1408811520255}}}]}}}}}}}},"size":0}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at
org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at
org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more
[2014-08-23 16:49:05,401][INFO ][node ] [Plunderer]
stopping ...
[2014-08-23 16:49:06,737][INFO ][node ] [Plunderer]
stopped
[2014-08-23 16:49:06,737][INFO ][node ] [Plunderer]
closing ...
[2014-08-23 16:49:06,772][INFO ][node ] [Plunderer]
closed
[2014-08-23 20:02:09,693][WARN ][common.jna ] Unable to lock
JVM memory (ENOMEM). This can result in part of the JVM being swapped out.
Increase RLIMIT_MEMLOCK or run elasticsearch as root.
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet Abdol]
version[1.2.2], pid[2715], build[9902f08/2014-07-09T12:02:32Z]
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet Abdol]
initializing ...
[2014-08-23 20:02:09,940][INFO ][plugins ] [Ahmet Abdol]
loaded [], sites [kopf]
[2014-08-23 20:02:14,692][INFO ][node ] [Ahmet Abdol]
initialized

any ideas??

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Z3D%2BE_MQ%3DrprB_t5K6-qz9WtCheNzM%3DmepGHimoCow_w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Eitan Vesely) #3

Thanks Mark,

auth.log doesnt show any login or sudo at the time of the elastic
stopping...
nothing else is running on that machine - it is a dedicated ES server.

what i did find in the auth log is that someone is trying to hack into the
system, yet i dont see how it got to do with elastic stopping?

On Sunday, August 24, 2014 4:35:41 AM UTC+3, Mark Walkom wrote:

Something is stopping the service.

If you are on linux check the auth log, if anyone is using sudo to stop it
then you will see that logged. Otherwise, what else runs on the machine?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 24 August 2014 06:15, Eitan Vesely <eita...@gmail.com <javascript:>>
wrote:

Hi Guys,
i've installed ES a month ago and its working just fine.

today, for some reason, ES just went down for no visible reason:

here is what i see in the log file :

[2014-08-23 16:47:11,272][DEBUG][action.search.type ] [Plunderer]
[g30nm0bi2j663tgu6ud][1], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad]
lastShard [true]
org.elasticsearch.search.SearchParseException: [g30nm0bi2j663tgu6ud][1]:
from[-1],size[-1]: Parse Failure [Failed to parse source
[{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"user_count","interval":"1h"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":""}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":1408811520255}}}]}}}}}}}},"size":0}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at
org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at
org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more
[2014-08-23 16:47:11,273][DEBUG][action.search.type ] [Plunderer]
[g30nm0bi2j663tgu6ud][0], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad]
org.elasticsearch.search.SearchParseException: [g30nm0bi2j663tgu6ud][0]:
from[-1],size[-1]: Parse Failure [Failed to parse source
[{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"user_count","interval":"1h"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"
"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":1408811520255}}}]}}}}}}}},"size":0}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at
org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at
org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more
[2014-08-23 16:49:05,401][INFO ][node ] [Plunderer]
stopping ...
[2014-08-23 16:49:06,737][INFO ][node ] [Plunderer]
stopped
[2014-08-23 16:49:06,737][INFO ][node ] [Plunderer]
closing ...
[2014-08-23 16:49:06,772][INFO ][node ] [Plunderer]
closed
[2014-08-23 20:02:09,693][WARN ][common.jna ] Unable to
lock JVM memory (ENOMEM). This can result in part of the JVM being swapped
out. Increase RLIMIT_MEMLOCK or run elasticsearch as root.
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet Abdol]
version[1.2.2], pid[2715], build[9902f08/2014-07-09T12:02:32Z]
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet Abdol]
initializing ...
[2014-08-23 20:02:09,940][INFO ][plugins ] [Ahmet Abdol]
loaded [], sites [kopf]
[2014-08-23 20:02:14,692][INFO ][node ] [Ahmet Abdol]
initialized

any ideas??

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d87b8ca2-8f11-4f5c-b0f1-d661d5bf2f3b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Eitan Vesely) #4

I did find the shutdown request in the syslog:

Aug 23 16:49:01 medisafelog2 kernel: [3361057.489168] hv_utils: Shutdown
request received - graceful shutdown initiated

yet i have no idea who or what initiated it... how can i dig in?

On Sunday, August 24, 2014 12:01:50 PM UTC+3, Eitan Vesely wrote:

Thanks Mark,

auth.log doesnt show any login or sudo at the time of the elastic
stopping...
nothing else is running on that machine - it is a dedicated ES server.

what i did find in the auth log is that someone is trying to hack into the
system, yet i dont see how it got to do with elastic stopping?

On Sunday, August 24, 2014 4:35:41 AM UTC+3, Mark Walkom wrote:

Something is stopping the service.

If you are on linux check the auth log, if anyone is using sudo to stop
it then you will see that logged. Otherwise, what else runs on the machine?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 24 August 2014 06:15, Eitan Vesely eita...@gmail.com wrote:

Hi Guys,
i've installed ES a month ago and its working just fine.

today, for some reason, ES just went down for no visible reason:

here is what i see in the log file :

[2014-08-23 16:47:11,272][DEBUG][action.search.type ] [Plunderer]
[g30nm0bi2j663tgu6ud][1], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad]
lastShard [true]
org.elasticsearch.search.SearchParseException: [g30nm0bi2j663tgu6ud][1]:
from[-1],size[-1]: Parse Failure [Failed to parse source
[{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"user_count","interval":"1h"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":""}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":1408811520255}}}]}}}}}}}},"size":0}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at
org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at
org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more
[2014-08-23 16:47:11,273][DEBUG][action.search.type ] [Plunderer]
[g30nm0bi2j663tgu6ud][0], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], s[STARTED]:
Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad]
org.elasticsearch.search.SearchParseException: [g30nm0bi2j663tgu6ud][0]:
from[-1],size[-1]: Parse Failure [Failed to parse source
[{"facets":{"0":{"date_histogram":{"key_field":"@timestamp","value_field":"user_count","interval":"1h"},"global":true,"facet_filter":{"fquery":{"query":{"filtered":{"query":{"query_string":{"query":"
"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":1408811520255}}}]}}}}}}}},"size":0}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:252)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at
org.elasticsearch.search.facet.datehistogram.DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at
org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
... 9 more
[2014-08-23 16:49:05,401][INFO ][node ] [Plunderer]
stopping ...
[2014-08-23 16:49:06,737][INFO ][node ] [Plunderer]
stopped
[2014-08-23 16:49:06,737][INFO ][node ] [Plunderer]
closing ...
[2014-08-23 16:49:06,772][INFO ][node ] [Plunderer]
closed
[2014-08-23 20:02:09,693][WARN ][common.jna ] Unable to
lock JVM memory (ENOMEM). This can result in part of the JVM being swapped
out. Increase RLIMIT_MEMLOCK or run elasticsearch as root.
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet
Abdol] version[1.2.2], pid[2715], build[9902f08/2014-07-09T12:02:32Z]
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet
Abdol] initializing ...
[2014-08-23 20:02:09,940][INFO ][plugins ] [Ahmet
Abdol] loaded [], sites [kopf]
[2014-08-23 20:02:14,692][INFO ][node ] [Ahmet
Abdol] initialized

any ideas??

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bfc73608-e6fe-489a-8f38-33f033b61b66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #5

What version of ES are you running?
Are you running on a hosting service and if so do you have a firewall
protecting the host - ie it's not open to the entire internet?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 24 August 2014 19:05, Eitan Vesely eitanjnk@gmail.com wrote:

I did find the shutdown request in the syslog:

Aug 23 16:49:01 medisafelog2 kernel: [3361057.489168] hv_utils: Shutdown
request received - graceful shutdown initiated

yet i have no idea who or what initiated it... how can i dig in?

On Sunday, August 24, 2014 12:01:50 PM UTC+3, Eitan Vesely wrote:

Thanks Mark,

auth.log doesnt show any login or sudo at the time of the elastic
stopping...
nothing else is running on that machine - it is a dedicated ES server.

what i did find in the auth log is that someone is trying to hack into
the system, yet i dont see how it got to do with elastic stopping?

On Sunday, August 24, 2014 4:35:41 AM UTC+3, Mark Walkom wrote:

Something is stopping the service.

If you are on linux check the auth log, if anyone is using sudo to stop
it then you will see that logged. Otherwise, what else runs on the machine?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 24 August 2014 06:15, Eitan Vesely eita...@gmail.com wrote:

Hi Guys,
i've installed ES a month ago and its working just fine.

today, for some reason, ES just went down for no visible reason:

here is what i see in the log file :

[2014-08-23 16:47:11,272][DEBUG][action.search.type ]
[Plunderer] [g30nm0bi2j663tgu6ud][1], node[Vc4xSuh1S1qQOvQdv-wD_A], [P],
s[STARTED]: Failed to execute [org.elasticsearch.action.
search.SearchRequest@5531dfad] lastShard [true]
org.elasticsearch.search.SearchParseException:
[g30nm0bi2j663tgu6ud][1]: from[-1],size[-1]: Parse Failure [Failed to parse
source [{"facets":{"0":{"date_histogram":{"key_field":"@
timestamp","value_field":"user_count","interval":"1h"},"
global":true,"facet_filter":{"fquery":{"query":{"filtered":{
"query":{"query_string":{"query":""}},"filter":{"bool":
{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":
1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":
1408811520255}}}]}}}}}}}},"size":0}]]
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:649)
at org.elasticsearch.search.SearchService.createContext(
SearchService.java:511)
at org.elasticsearch.search.SearchService.createAndPutContext(
SearchService.java:483)
at org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:252)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:206)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:203)
at org.elasticsearch.search.action.SearchServiceTransportAction$
23.run(SearchServiceTransportAction.java:517)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at org.elasticsearch.search.facet.datehistogram.
DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at org.elasticsearch.search.facet.FacetParseElement.parse(
FacetParseElement.java:93)
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:633)
... 9 more
[2014-08-23 16:47:11,273][DEBUG][action.search.type ]
[Plunderer] [g30nm0bi2j663tgu6ud][0], node[Vc4xSuh1S1qQOvQdv-wD_A], [P],
s[STARTED]: Failed to execute [org.elasticsearch.action.
search.SearchRequest@5531dfad]
org.elasticsearch.search.SearchParseException:
[g30nm0bi2j663tgu6ud][0]: from[-1],size[-1]: Parse Failure [Failed to parse
source [{"facets":{"0":{"date_histogram":{"key_field":"@
timestamp","value_field":"user_count","interval":"1h"},"
global":true,"facet_filter":{"fquery":{"query":{"filtered":{
"query":{"query_string":{"query":"
"}},"filter":{"bool":
{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":
1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":
1408811520255}}}]}}}}}}}},"size":0}]]
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:649)
at org.elasticsearch.search.SearchService.createContext(
SearchService.java:511)
at org.elasticsearch.search.SearchService.createAndPutContext(
SearchService.java:483)
at org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:252)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:206)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:203)
at org.elasticsearch.search.action.SearchServiceTransportAction$
23.run(SearchServiceTransportAction.java:517)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at org.elasticsearch.search.facet.datehistogram.
DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at org.elasticsearch.search.facet.FacetParseElement.parse(
FacetParseElement.java:93)
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:633)
... 9 more
[2014-08-23 16:49:05,401][INFO ][node ] [Plunderer]
stopping ...
[2014-08-23 16:49:06,737][INFO ][node ] [Plunderer]
stopped
[2014-08-23 16:49:06,737][INFO ][node ] [Plunderer]
closing ...
[2014-08-23 16:49:06,772][INFO ][node ] [Plunderer]
closed
[2014-08-23 20:02:09,693][WARN ][common.jna ] Unable to
lock JVM memory (ENOMEM). This can result in part of the JVM being swapped
out. Increase RLIMIT_MEMLOCK or run elasticsearch as root.
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet
Abdol] version[1.2.2], pid[2715], build[9902f08/2014-07-09T12:02:32Z]
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet
Abdol] initializing ...
[2014-08-23 20:02:09,940][INFO ][plugins ] [Ahmet
Abdol] loaded [], sites [kopf]
[2014-08-23 20:02:14,692][INFO ][node ] [Ahmet
Abdol] initialized

any ideas??

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bfc73608-e6fe-489a-8f38-33f033b61b66%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bfc73608-e6fe-489a-8f38-33f033b61b66%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZmqUgi4uJV6wzjvfM9YWxvDY9xkfgCHJFSRm_6D5vDUQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #6

The company which is providing the hosting service had shut down the
virtual machine, "hv_utils" is a message from the hypervisor.

This is not related to Elasticsearch at all.

Jörg

On Sun, Aug 24, 2014 at 11:19 AM, Mark Walkom markw@campaignmonitor.com
wrote:

What version of ES are you running?
Are you running on a hosting service and if so do you have a firewall
protecting the host - ie it's not open to the entire internet?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 24 August 2014 19:05, Eitan Vesely eitanjnk@gmail.com wrote:

I did find the shutdown request in the syslog:

Aug 23 16:49:01 medisafelog2 kernel: [3361057.489168] hv_utils: Shutdown
request received - graceful shutdown initiated

yet i have no idea who or what initiated it... how can i dig in?

On Sunday, August 24, 2014 12:01:50 PM UTC+3, Eitan Vesely wrote:

Thanks Mark,

auth.log doesnt show any login or sudo at the time of the elastic
stopping...
nothing else is running on that machine - it is a dedicated ES server.

what i did find in the auth log is that someone is trying to hack into
the system, yet i dont see how it got to do with elastic stopping?

On Sunday, August 24, 2014 4:35:41 AM UTC+3, Mark Walkom wrote:

Something is stopping the service.

If you are on linux check the auth log, if anyone is using sudo to stop
it then you will see that logged. Otherwise, what else runs on the machine?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 24 August 2014 06:15, Eitan Vesely eita...@gmail.com wrote:

Hi Guys,
i've installed ES a month ago and its working just fine.

today, for some reason, ES just went down for no visible reason:

here is what i see in the log file :

[2014-08-23 16:47:11,272][DEBUG][action.search.type ]
[Plunderer] [g30nm0bi2j663tgu6ud][1], node[Vc4xSuh1S1qQOvQdv-wD_A], [P],
s[STARTED]: Failed to execute [org.elasticsearch.action.
search.SearchRequest@5531dfad] lastShard [true]
org.elasticsearch.search.SearchParseException:
[g30nm0bi2j663tgu6ud][1]: from[-1],size[-1]: Parse Failure [Failed to parse
source [{"facets":{"0":{"date_histogram":{"key_field":"@
timestamp","value_field":"user_count","interval":"1h"},"
global":true,"facet_filter":{"fquery":{"query":{"filtered":{
"query":{"query_string":{"query":""}},"filter":{"bool":
{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":
1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":
1408811520255}}}]}}}}}}}},"size":0}]]
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:649)
at org.elasticsearch.search.SearchService.createContext(
SearchService.java:511)
at org.elasticsearch.search.SearchService.createAndPutContext(
SearchService.java:483)
at org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:252)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:206)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:203)
at org.elasticsearch.search.action.SearchServiceTransportAction$
23.run(SearchServiceTransportAction.java:517)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at org.elasticsearch.search.facet.datehistogram.
DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at org.elasticsearch.search.facet.FacetParseElement.parse(
FacetParseElement.java:93)
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:633)
... 9 more
[2014-08-23 16:47:11,273][DEBUG][action.search.type ]
[Plunderer] [g30nm0bi2j663tgu6ud][0], node[Vc4xSuh1S1qQOvQdv-wD_A], [P],
s[STARTED]: Failed to execute [org.elasticsearch.action.
search.SearchRequest@5531dfad]
org.elasticsearch.search.SearchParseException:
[g30nm0bi2j663tgu6ud][0]: from[-1],size[-1]: Parse Failure [Failed to parse
source [{"facets":{"0":{"date_histogram":{"key_field":"@
timestamp","value_field":"user_count","interval":"1h"},"
global":true,"facet_filter":{"fquery":{"query":{"filtered":{
"query":{"query_string":{"query":"
"}},"filter":{"bool":
{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":
1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":
1408811520255}}}]}}}}}}}},"size":0}]]
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:649)
at org.elasticsearch.search.SearchService.createContext(
SearchService.java:511)
at org.elasticsearch.search.SearchService.createAndPutContext(
SearchService.java:483)
at org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:252)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:206)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:203)
at org.elasticsearch.search.action.SearchServiceTransportAction$
23.run(SearchServiceTransportAction.java:517)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at org.elasticsearch.search.facet.datehistogram.
DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at org.elasticsearch.search.facet.FacetParseElement.parse(
FacetParseElement.java:93)
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:633)
... 9 more
[2014-08-23 16:49:05,401][INFO ][node ]
[Plunderer] stopping ...
[2014-08-23 16:49:06,737][INFO ][node ]
[Plunderer] stopped
[2014-08-23 16:49:06,737][INFO ][node ]
[Plunderer] closing ...
[2014-08-23 16:49:06,772][INFO ][node ]
[Plunderer] closed
[2014-08-23 20:02:09,693][WARN ][common.jna ] Unable to
lock JVM memory (ENOMEM). This can result in part of the JVM being swapped
out. Increase RLIMIT_MEMLOCK or run elasticsearch as root.
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet
Abdol] version[1.2.2], pid[2715], build[9902f08/2014-07-09T12:02:32Z]
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet
Abdol] initializing ...
[2014-08-23 20:02:09,940][INFO ][plugins ] [Ahmet
Abdol] loaded [], sites [kopf]
[2014-08-23 20:02:14,692][INFO ][node ] [Ahmet
Abdol] initialized

any ideas??

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bfc73608-e6fe-489a-8f38-33f033b61b66%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bfc73608-e6fe-489a-8f38-33f033b61b66%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZmqUgi4uJV6wzjvfM9YWxvDY9xkfgCHJFSRm_6D5vDUQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZmqUgi4uJV6wzjvfM9YWxvDY9xkfgCHJFSRm_6D5vDUQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFJqbLXEF_rsY9zLzp3bwfBmawjRZKZ5VJ5GNmGsnuZ1g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Eitan Vesely) #7

Thanks Jorg!

I'll take it up to the hosting company... (Azure!)

On Sunday, August 24, 2014 3:52:54 PM UTC+3, Jörg Prante wrote:

The company which is providing the hosting service had shut down the
virtual machine, "hv_utils" is a message from the hypervisor.

This is not related to Elasticsearch at all.

Jörg

On Sun, Aug 24, 2014 at 11:19 AM, Mark Walkom <ma...@campaignmonitor.com
<javascript:>> wrote:

What version of ES are you running?
Are you running on a hosting service and if so do you have a firewall
protecting the host - ie it's not open to the entire internet?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 24 August 2014 19:05, Eitan Vesely <eita...@gmail.com <javascript:>>
wrote:

I did find the shutdown request in the syslog:

Aug 23 16:49:01 medisafelog2 kernel: [3361057.489168] hv_utils: Shutdown
request received - graceful shutdown initiated

yet i have no idea who or what initiated it... how can i dig in?

On Sunday, August 24, 2014 12:01:50 PM UTC+3, Eitan Vesely wrote:

Thanks Mark,

auth.log doesnt show any login or sudo at the time of the elastic
stopping...
nothing else is running on that machine - it is a dedicated ES server.

what i did find in the auth log is that someone is trying to hack into
the system, yet i dont see how it got to do with elastic stopping?

On Sunday, August 24, 2014 4:35:41 AM UTC+3, Mark Walkom wrote:

Something is stopping the service.

If you are on linux check the auth log, if anyone is using sudo to
stop it then you will see that logged. Otherwise, what else runs on the
machine?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 24 August 2014 06:15, Eitan Vesely eita...@gmail.com wrote:

Hi Guys,
i've installed ES a month ago and its working just fine.

today, for some reason, ES just went down for no visible reason:

here is what i see in the log file :

[2014-08-23 16:47:11,272][DEBUG][action.search.type ]
[Plunderer] [g30nm0bi2j663tgu6ud][1], node[Vc4xSuh1S1qQOvQdv-wD_A], [P],
s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@5531dfad]
lastShard [true]
org.elasticsearch.search.SearchParseException:
[g30nm0bi2j663tgu6ud][1]: from[-1],size[-1]: Parse Failure [Failed to parse
source [{"facets":{"0":{"date_histogram":{"key_field":"@
timestamp","value_field":"user_count","interval":"1h"},"
global":true,"facet_filter":{"fquery":{"query":{"filtered":{
"query":{"query_string":{"query":""}},"filter":{"bool":
{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":
1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":
1408811520255}}}]}}}}}}}},"size":0}]]
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:649)
at org.elasticsearch.search.SearchService.createContext(
SearchService.java:511)
at org.elasticsearch.search.SearchService.createAndPutContext(
SearchService.java:483)
at org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:252)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:206)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:203)
at org.elasticsearch.search.action.SearchServiceTransportAction$
23.run(SearchServiceTransportAction.java:517)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at org.elasticsearch.search.facet.datehistogram.
DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at org.elasticsearch.search.facet.FacetParseElement.parse(
FacetParseElement.java:93)
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:633)
... 9 more
[2014-08-23 16:47:11,273][DEBUG][action.search.type ]
[Plunderer] [g30nm0bi2j663tgu6ud][0], node[Vc4xSuh1S1qQOvQdv-wD_A], [P],
s[STARTED]: Failed to execute [org.elasticsearch.action.
search.SearchRequest@5531dfad]
org.elasticsearch.search.SearchParseException:
[g30nm0bi2j663tgu6ud][0]: from[-1],size[-1]: Parse Failure [Failed to parse
source [{"facets":{"0":{"date_histogram":{"key_field":"@
timestamp","value_field":"user_count","interval":"1h"},"
global":true,"facet_filter":{"fquery":{"query":{"filtered":{
"query":{"query_string":{"query":"
"}},"filter":{"bool":
{"must":[{"range":{"@timestamp":{"from":1407602785182,"to":
1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to":
1408811520255}}}]}}}}}}}},"size":0}]]
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:649)
at org.elasticsearch.search.SearchService.createContext(
SearchService.java:511)
at org.elasticsearch.search.SearchService.createAndPutContext(
SearchService.java:483)
at org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:252)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:206)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:203)
at org.elasticsearch.search.action.SearchServiceTransportAction$
23.run(SearchServiceTransportAction.java:517)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException:
Facet [0]: (key) field [@timestamp] not found
at org.elasticsearch.search.facet.datehistogram.
DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160)
at org.elasticsearch.search.facet.FacetParseElement.parse(
FacetParseElement.java:93)
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:633)
... 9 more
[2014-08-23 16:49:05,401][INFO ][node ]
[Plunderer] stopping ...
[2014-08-23 16:49:06,737][INFO ][node ]
[Plunderer] stopped
[2014-08-23 16:49:06,737][INFO ][node ]
[Plunderer] closing ...
[2014-08-23 16:49:06,772][INFO ][node ]
[Plunderer] closed
[2014-08-23 20:02:09,693][WARN ][common.jna ] Unable to
lock JVM memory (ENOMEM). This can result in part of the JVM being swapped
out. Increase RLIMIT_MEMLOCK or run elasticsearch as root.
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet
Abdol] version[1.2.2], pid[2715], build[9902f08/2014-07-09T12:02:32Z]
[2014-08-23 20:02:09,921][INFO ][node ] [Ahmet
Abdol] initializing ...
[2014-08-23 20:02:09,940][INFO ][plugins ] [Ahmet
Abdol] loaded [], sites [kopf]
[2014-08-23 20:02:14,692][INFO ][node ] [Ahmet
Abdol] initialized

any ideas??

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bfc73608-e6fe-489a-8f38-33f033b61b66%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bfc73608-e6fe-489a-8f38-33f033b61b66%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZmqUgi4uJV6wzjvfM9YWxvDY9xkfgCHJFSRm_6D5vDUQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZmqUgi4uJV6wzjvfM9YWxvDY9xkfgCHJFSRm_6D5vDUQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/265cb108-5644-4798-a70c-db067e6530d2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #8