Elasticsearch shuts down with no reason

Elastic Stack Elasticsearch
1

Hi
We're running Elasticsearch 1.7.2 on ubuntu 14.04, the system has enough memory and other services on the server work just fine. But sometimes elastic shutsdown without any thing logged to it's log, or anything logged to syslog. How can I investigate this issue?
This is part of the log that shows when it was down:

[2015-11-23 07:47:07,757][INFO ][com.asquera.elasticsearch.plugins.http.HttpBasicServer] [Pisces] Authorization:xx, Host:xx.xx.xx.xx:yy, Path:/, :null, Request-IP:xx.xx.xx.xx:yy, Client-IP:null, X-Client-IPnull
[2015-11-23 11:27:31,948][INFO ][node                     ] [zzz] version[1.7.2], pid[30103], build[e43676b/2015-09-14T09:49:53Z]
[2015-11-23 11:27:31,949][INFO ][node                     ] [zzz] initializing ...
[2015-11-23 11:27:32,150][INFO ][plugins                  ] [zzz] loaded [http-basic-server-plugin], sites []

note that there is a 4 hour gap in there, and there is no log related to shut down.
Thanks.

2

you could try to set the loglevel to DEBUG or TRACE, depending on what you set it to now.

TRACE might be overkill, as LOT will be logged

3

Check your OS logs as well, maybe the oomkiller got it?

4

I'm setting it to DEBUG, and I will wait to see if solves it.
I don't know were to look beside syslog?

5

Now this is when it shutsdown with DEBUG level of log:

[2015-11-24 02:10:54,121][DEBUG][index.shard              ] [Keen Marlow] [haystack_backup][4] updating index_buffer_size from [64mb] to
 [4mb]
[2015-11-24 02:10:54,121][DEBUG][index.shard              ] [Keen Marlow] [product_index_backup][0] updating index_buffer_size from [64m
b] to [4mb]
[2015-11-24 02:10:54,121][DEBUG][index.shard              ] [Keen Marlow] [product_index_backup][1] updating index_buffer_size from [64m
b] to [4mb]
[2015-11-24 02:10:54,121][DEBUG][index.shard              ] [Keen Marlow] [product_index_backup][2] updating index_buffer_size from [64m
b] to [4mb]
[2015-11-24 02:10:54,121][DEBUG][index.shard              ] [Keen Marlow] [product_index_backup][3] updating index_buffer_size from [64m
b] to [4mb]
[2015-11-24 02:10:54,121][DEBUG][index.shard              ] [Keen Marlow] [product_index_backup][4] updating index_buffer_size from [64m
b] to [4mb]
[2015-11-24 02:11:24,015][INFO ][com.asquera.elasticsearch.plugins.http.HttpBasicServer] [Keen Marlow] Authorization:xx, Host:xx.xx.xx.xx:yy, Path:/, :null, Request-IP:xx.xx.xx.xx, Client-IP:null, X-Client-IPnull
[2015-11-24 02:12:15,416][INFO ][com.asquera.elasticsearch.plugins.http.HttpBasicServer] [Keen Marlow] Authorization:Basic xxxx, Host:xx.xx.xx.xx:yy, Path:/product_index/product/webservice.ProductPage.282933/, :null, Request-IP:xx.xx.xx.xx
, Client-IP:null, X-Client-IPnull
[2015-11-24 02:25:18,402][INFO ][node                     ] [Stakar] version[1.7.2], pid[16597], build[e43676b/2015-09-14T09:49:53Z]
[2015-11-24 02:25:18,402][INFO ][node                     ] [Stakar] initializing ...
[2015-11-24 02:25:18,403][DEBUG][node                     ] [Stakar] using home [/usr/share/elasticsearch], config [/etc/elasticsearch],
 data [[/var/lib/elasticsearch]], logs [/var/log/elasticsearch], work [/tmp/elasticsearch], plugins [/usr/share/elasticsearch/plugins]
[2015-11-24 02:25:18,550][DEBUG][plugins                  ] [Stakar] lucene property is not set in plugin es-plugin.properties file. Ski
pping test.
[2015-11-24 02:25:18,551][DEBUG][plugins                  ] [Stakar] [/usr/share/elasticsearch/plugins/http-basic-server-plugin/_site] d
irectory does not exist.
[2015-11-24 02:25:18,552][DEBUG][plugins                  ] [Stakar] [/usr/share/elasticsearch/plugins/http-basic/_site] directory does
not exist.
[2015-11-24 02:25:18,553][INFO ][plugins                  ] [Stakar] loaded [http-basic-server-plugin], sites []
[2015-11-24 02:25:18,581][DEBUG][common.compress.lzf      ] using encoder [VanillaChunkDecoder] and decoder[{}]
[2015-11-24 02:25:18,605][DEBUG][env                      ] [Stakar] using node location [[NodePath{path=/var/lib/elasticsearch/torob/no
des/0}]], local_node_id [0]

notice the 13 minute gap between 2:12 & 2:25, when I started elastic again.
Do you see anything in these logs?

6

Does this have any relation to issue 14188?

7

Maybe it's the asquera plugin?

8

What about it? is it buggy?

9

I have no idea, have you tried without it?