ElasticSearch stop after few days

Hi everyone,

I installed
ElasticSearch Version: 2.1.0, Build: 72cd1f1/2015-11-18T22:40:03Z, JVM: 1.7.0_79
logstash 2.1.0
on CentOS Linux release 7.1.1503 (Core) with "Shield" plugin successfully but
after few days elasticsearch crashed and it is not starting.

[a@server ~]service elasticsearch status
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
Active: failed (Result: exit-code) since Sat 2015-12-10 03:07:22 EST; 6mint ago
Docs: http://www.elastic.co
Process: 14633 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 14631 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 14633 (code=exited, status=1/FAILURE)

Dec 10 03:07:21 server systemd[1]: Starting Elasticsearch...
Dec 10 03:07:21 server systemd[1]: Started Elasticsearch.
Dec 10 03:07:22 server systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Dec 10 03:07:22 server systemd[1]: Unit elasticsearch.service entered failed state.

I run command

[a@server ~] journalctl -u elasticsearch.service
-- Logs begin at Wed 2015-12-02 13:56:14 EST, end at Sat 2015-12-12 04:59:49 EST. --
Dec 02 13:56:47 server systemd[1]: Starting Elasticsearch...
Dec 02 13:56:47 server systemd[1]: Started Elasticsearch.
Dec 03 01:22:05 server systemd[1]: Stopping Elasticsearch...
Dec 03 01:22:05 server systemd[1]: Starting Elasticsearch...
Dec 03 01:22:05 server systemd[1]: Started Elasticsearch.
Dec 10 03:07:21 server systemd[1]: Starting Elasticsearch...
Dec 10 03:07:21 server systemd[1]: Started Elasticsearch.
Dec 10 03:07:22 server systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Dec 10 03:07:22 server systemd[1]: Unit elasticsearch.service entered failed state.

Last log messages are

[2015-12-11 06:27:19,895][ERROR][license.plugin.core ] [gznw]

License will expire on [Thursday, January 07, 2016]. If you have a new license, please update it.

Otherwise, please reach out to your support contact.

Commercial plugins operate with reduced functionality on license expiration:

- shield

- Cluster health, cluster stats and indices stats operations are blocked

- All data operations (read and write) continue to work

[2015-12-09 09:42:28,581][INFO ][cluster.metadata ] [gznw] [logstash-2015.12.11] create_mapping [logs]
[2015-12-09 10:02:35,391][INFO ][cluster.metadata ] [gznw] [logstash-2015.12.11] update_mapping [logs]
[2015-12-09 10:43:51,927][INFO ][rest.suppressed ] /logstash-/_mapping/field/* Params: {index=logstash-, allow_no_indices=false, include_defaults=true, _=1449848632840, fields=, ignore_unavailable=false}
[logstash-] IndexNotFoundException[no such index]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566)
...
[2015-12-09 10:43:59,773][INFO ][rest.suppressed ] /logstash-2015.12./_mapping/field/
Params: {index=logstash-2015.12., allow_no_indices=false, include_defaults=true, _=1449848640723, fields=*, ignore_unavailable=false}
[logstash-2015.12.] IndexNotFoundException[no such index]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566)
...
[2015-12-09 10:52:00,925][WARN ][shield.transport.netty ] [gznw] received plaintext http traffic on a https channel, closing connection [id: 0x92f71730, /120.210.205.233:2854 => /192.168.0.1:9200]

Now elasticsearch is not starting. I search in form and in google but can't find my solution.

Are those all of the recent logs? I don't see anything that would indicate an ES failure or a crash.

Is the machine critically low on memory? I suppose it's possible that the Linux OOM Killer killed ES without notice. Please check syslog and kern.log around the time when Elasticsearch went down - do you see anything about oom-killer in there?

Also note that your trial license of Shield has expired, so as the error message states, all cluster health APIs are disabled.

Yes, It was linux who kills the ES and kibana occupies the memory. I stopped kibana then started ES only last INDEX corrupted. Now ES is running.
Thanks "Steve Kearns"