Elasticsearch failed to start

Elastic Stack Elasticsearch
#1

Hi,

My elasticsearch instance was working just fine, I made some changes to elasticsearch.yml file for email setting and restarted service, but elasticsearch failed to start and I could not even connect to elasticsearch using curl on by ubuntu machine it says failed to connect both on localhost and on IP.
I removed changes and restart service but still it failed to start.

elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2020-09-03 19:50:52 PKT; 2min 18s ago
 Docs: https://www.elastic.co
  Process: 948 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/F
 Main PID: 948 (code=exited, status=1/FAILURE)

ستمبر 03 19:50:52 elk-rewtez systemd-entrypoint[948]: Error: Could not create the Java Virtual Machine.
ستمبر 03 19:50:52 elk-rewtez systemd-entrypoint[948]: Error: A fatal exception has occurred. Program will exit.
ستمبر 03 19:50:52 elk-rewtez systemd-entrypoint[948]:         at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics
ستمبر 03 19:50:52 elk-rewtez systemd-entrypoint[948]:         at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgon
ستمبر 03 19:50:52 elk-rewtez systemd-entrypoint[948]:         at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.jav
ستمبر 03 19:50:52 elk-rewtez systemd-entrypoint[948]:         at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptions
ستمبر 03 19:50:52 elk-rewtez systemd-entrypoint[948]:         at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser
ستمبر 03 19:50:52 elk-rewtez systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
ستمبر 03 19:50:52 elk-rewtez systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
ستمبر 03 19:50:52 elk-rewtez systemd[1]: Failed to start Elasticsearch.

can someone help here

#2

Can you share your elasticsearch log file ?

#3 

[2020-09-03T11:02:30,015][WARN ][o.e.x.r.j.RollupJobTask  ] [node-1] Rollup job [windows] failed with an exception:
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:403) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.TransportService$6.handleException(TransportService.java:639) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1173) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1282) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1256) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:413) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.0.jar:7.8.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-09-03T11:03:00,004][WARN ][o.e.x.r.j.RollupJobTask  ] [node-1] Rollup job [windows] failed with an exception:
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:582) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearcroot@elk-rewtez:/mnt/sdb200G/logs/elasticsearch#
#4

there are only 2 warning, this will not block your node from starting,
can you provide fulle log file or look at latests ERROR when you tried to restart your node

1 Like
#5

Sure looks like you are failing on illegal flags on the JVM as JVM won't even start in your first log set (so can't see how you can get second log set at all, which is after node running).

Could not create the Java Virtual Machine.
JvmOptionsParser / flagsFinal

Seems you messed up the elasticsearch.yml file or touched the JVM options file.

#6

these are logs which contains error, but they are from 10 am i changed yml file @ arround 4 pm

[2020-09-03T10:59:29,878][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-9758653434182231128, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
#7

I did not change any thing in JVM options file, just changed some setting in elasticsearch.yml

#8

Suggest post full elasticsearch.yml file and jvm options file. As snippets.

Also look for ERROR in log as @ylasri suggests.