Hi,
I'm running an elasticsearch server 6.3 and i've upgraded it to 6.4.2.
After the upgrade i'm running into the problem that service after a few minutes stops and fails.
I can restart it without a problem
When looking at the incides:
- http:///_cat/indices?v
Are being set from red to yellow.
Afterwards this all indices have become yellow Kibana starts to function.
service log: (i've had to shorten it after being limit to 7000 characters)
Oct 30 15:15:04 ip-10-150-34-188 systemd[1]: Started Elasticsearch.
Oct 30 15:15:04 ip-10-150-34-188 systemd[1]: Starting Elasticsearch...
Oct 30 15:15:04 ip-10-150-34-188 elasticsearch[3227]: Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Oct 30 15:20:37 ip-10-150-34-188 systemd[1]: elasticsearch.service: main process exited, code=killed, status=6/ABRT
Oct 30 15:20:37 ip-10-150-34-188 systemd[1]: Unit elasticsearch.service entered failed state.
Oct 30 15:20:37 ip-10-150-34-188 systemd[1]: elasticsearch.service failed.
Elasticsearch log:
[2018-10-30T15:18:16,257][DEBUG][o.e.a.s.TransportSearchAction] [] All shards failed for phase: [query]
[2018-10-30T15:18:16,258][WARN ][r.suppressed ] path: /.kibana/_search, params: {ignore_unavailable=true, index=.kibana, filter_path=aggregations.types.buckets}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) [elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) [elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) [elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) [elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:210) [elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:189) [elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.4.2.jar:6.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.2.jar:6.4.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:844) [?:?]
[2018-10-30T15:18:18,559][INFO ][o.e.m.j.JvmGcMonitorService] [] [gc][177] overhead, spent [267ms] collecting in the last [1s]
I'm running it on an VM in AWS.
AMI: AWS Linux v2 (Red Hat)
Type: m5.xlarge
I'm a bit of a noob on the elasticsearch topic so please forgive me if i'm missing something obvious.
Can someone help me out or tell me what i'm doing or what is going wrong!?
Many thanks in advance.