ElasticSearch stop after few days


(Zulquarnain) #1

Hi everyone,

I installed
ElasticSearch Version: 2.1.0, Build: 72cd1f1/2015-11-18T22:40:03Z, JVM: 1.7.0_79
logstash 2.1.0
on CentOS Linux release 7.1.1503 (Core) with "Shield" plugin successfully but
after few days elasticsearch crashed and it is not starting.

[a@server ~]service elasticsearch status
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled)
Active: failed (Result: exit-code) since Sat 2015-12-10 03:07:22 EST; 6mint ago
Docs: http://www.elastic.co
Process: 14633 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${DATA_DIR} -Des.default.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 14631 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 14633 (code=exited, status=1/FAILURE)

Dec 10 03:07:21 server systemd[1]: Starting Elasticsearch...
Dec 10 03:07:21 server systemd[1]: Started Elasticsearch.
Dec 10 03:07:22 server systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Dec 10 03:07:22 server systemd[1]: Unit elasticsearch.service entered failed state.

I run command

[a@server ~] journalctl -u elasticsearch.service
-- Logs begin at Wed 2015-12-02 13:56:14 EST, end at Sat 2015-12-12 04:59:49 EST. --
Dec 02 13:56:47 server systemd[1]: Starting Elasticsearch...
Dec 02 13:56:47 server systemd[1]: Started Elasticsearch.
Dec 03 01:22:05 server systemd[1]: Stopping Elasticsearch...
Dec 03 01:22:05 server systemd[1]: Starting Elasticsearch...
Dec 03 01:22:05 server systemd[1]: Started Elasticsearch.
Dec 10 03:07:21 server systemd[1]: Starting Elasticsearch...
Dec 10 03:07:21 server systemd[1]: Started Elasticsearch.
Dec 10 03:07:22 server systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Dec 10 03:07:22 server systemd[1]: Unit elasticsearch.service entered failed state.

Last log messages are

[2015-12-11 06:27:19,895][ERROR][license.plugin.core ] [gznw]

License will expire on [Thursday, January 07, 2016]. If you have a new license, please update it.

Otherwise, please reach out to your support contact.

Commercial plugins operate with reduced functionality on license expiration:

- shield

- Cluster health, cluster stats and indices stats operations are blocked

- All data operations (read and write) continue to work

[2015-12-09 09:42:28,581][INFO ][cluster.metadata ] [gznw] [logstash-2015.12.11] create_mapping [logs]
[2015-12-09 10:02:35,391][INFO ][cluster.metadata ] [gznw] [logstash-2015.12.11] update_mapping [logs]
[2015-12-09 10:43:51,927][INFO ][rest.suppressed ] /logstash-/_mapping/field/* Params: {index=logstash-, allow_no_indices=false, include_defaults=true, _=1449848632840, fields=, ignore_unavailable=false}
[logstash-] IndexNotFoundException[no such index]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566)
...
[2015-12-09 10:43:59,773][INFO ][rest.suppressed ] /logstash-2015.12./_mapping/field/
Params: {index=logstash-2015.12., allow_no_indices=false, include_defaults=true, _=1449848640723, fields=*, ignore_unavailable=false}
[logstash-2015.12.] IndexNotFoundException[no such index]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566)
...
[2015-12-09 10:52:00,925][WARN ][shield.transport.netty ] [gznw] received plaintext http traffic on a https channel, closing connection [id: 0x92f71730, /120.210.205.233:2854 => /192.168.0.1:9200]

Now elasticsearch is not starting. I search in form and in google but can't find my solution.


(Steve Kearns) #2

Are those all of the recent logs? I don't see anything that would indicate an ES failure or a crash.

Is the machine critically low on memory? I suppose it's possible that the Linux OOM Killer killed ES without notice. Please check syslog and kern.log around the time when Elasticsearch went down - do you see anything about oom-killer in there?

Also note that your trial license of Shield has expired, so as the error message states, all cluster health APIs are disabled.


(Zulquarnain) #3

Yes, It was linux who kills the ES and kibana occupies the memory. I stopped kibana then started ES only last INDEX corrupted. Now ES is running.
Thanks "Steve Kearns"


(system) #4