# Elasticsearch fails to start on VM Boot

**URL:** <https://discuss.elastic.co/t/elasticsearch-fails-to-start-on-vm-boot/260645>\
**Category:** Elasticsearch\
**Created:** [January 10, 2021, 2:37pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-start-on-vm-boot/260645 "2021-01-10T14:37:06Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![droidus](https://avatars.discourse-cdn.com/v4/letter/d/13edae/32.png) [@droidus](https://discuss.elastic.co/u/droidus)\
**Post date:** [January 10, 2021, 2:37pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-start-on-vm-boot/260645/1 "2021-01-10T14:37:06Z")

</div>

I am running Ubuntu 20. Upon starting my machine, kibana fails, saying it can't connect to localhost:9200. I check on Elasticsearch, and I see this:

`● elasticsearch.service - Elasticsearch Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Sun 2021-01-10 09:32:32 EST; 1min 42s ago Docs: https://www.elastic.co Process: 824 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=143) Main PID: 824 (code=exited, status=143) Jan 10 09:31:16 elk systemd[1]: Starting Elasticsearch... Jan 10 09:32:31 elk systemd[1]: elasticsearch.service: start operation timed out. Terminating. Jan 10 09:32:32 elk systemd[1]: elasticsearch.service: Failed with result 'timeout'. Jan 10 09:32:32 elk systemd[1]: Failed to start Elasticsearch.`

And upon shutdown, it gets stuck here for some reason:  
 ![2021-01-10_09-29](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b45333e7f043ed8e97f9821b2b914524b90649b1.png) .  
I have to force shut it down.

I do a service elasticsearch start, and it starts up just fine, and everything is resolved. But upon a reboot, it does the same thing again.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 11, 2021, 12:07am UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-start-on-vm-boot/260645/2 "2021-01-11T00:07:53Z")

</div>

Please don't post pictures of text, they are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them 🙂

Can you directly check the Elasticsearch logs?

---

<div class="post-metadata">

**Author:** ![droidus](https://avatars.discourse-cdn.com/v4/letter/d/13edae/32.png) [@droidus](https://discuss.elastic.co/u/droidus)\
**Post date:** [January 12, 2021, 12:32pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-start-on-vm-boot/260645/3 "2021-01-12T12:32:27Z")

</div>

Sure, I am checking elasticsearch.log. I did a reboot. But I don't see any errors after it being stopped, then started. But it still fails to come up.  
Here are some errors that I did find:

```auto
[2021-01-12T07:18:35,149][INFO][o.e.x.i.IndexLifecycleRunner] [elk] policy [filebeat] for index [filebeat-7.10.0-2020.12.06] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [1489]
[2021-01-12T07:18:35,149][INFO][o.e.x.i.IndexLifecycleRunner] [elk] policy [filebeat] for index [filebeat-7.10.0-2020.12.18] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [692]
[2021-01-12T07:18:35,150][ERROR][o.e.x.i.IndexLifecycleRunner] [elk] policy [filebeat] for index [filebeat-7.10.0-2021.01.02] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [filebeat-7.10.0] does not point to index [filebeat-7.10.0-2021.01.02]
        at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:114) [x-pack-core-7.10.1.jar:7.10.1]
        at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:174) [x-pack-ilm-7.10.1.jar:7.10.1]
        at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:327) [x-pack-ilm-7.10.1.jar:7.10.1]
        at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:265) [x-pack-ilm-7.10.1.jar:7.10.1]
        at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.10.1.jar:7.10.1]
        at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:216) [x-pack-core-7.10.1.jar:7.10.1]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-01-12T07:18:35,151][ERROR][o.e.x.i.IndexLifecycleRunner] [elk] policy [filebeat] for index [filebeat-7.10.0-2021.01.08] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [filebeat-7.10.0] does not point to index [filebeat-7.10.0-2021.01.08]
        at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:114) [x-pack-core-7.10.1.jar:7.10.1]
        at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:174) [x-pack-ilm-7.10.1.jar:7.10.1]
        at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:327) [x-pack-ilm-7.10.1.jar:7.10.1]
        at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:265) [x-pack-ilm-7.10.1.jar:7.10.1]
        at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.10.1.jar:7.10.1]
        at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:216) [x-pack-core-7.10.1.jar:7.10.1]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

```

I assume this is not related to startup of the service though.

Here is after the reboot:

```auto
[2021-01-12T07:20:20,450][INFO][o.e.n.Node] [elk] stopped
[2021-01-12T07:20:20,451][INFO][o.e.n.Node] [elk] closing ...
[2021-01-12T07:20:20,532][INFO][o.e.n.Node] [elk] closed
[2021-01-12T07:21:49,176][INFO][o.e.n.Node] [elk] version[7.10.1], pid[800], build[default/deb/1c34507e66d7db1211f66f3513706fdf548736aa/2020-12-05T01:00:33.671820Z], OS[Linux/5.4.0-60-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-01-12T07:21:49,213][INFO][o.e.n.Node] [elk] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-01-12T07:21:49,214][INFO][o.e.n.Node] [elk] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-17815586022187814373, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-01-12T07:22:16,043][INFO][o.e.p.PluginsService] [elk] loaded module [aggs-matrix-stats]
[2021-01-12T07:22:16,045][INFO][o.e.p.PluginsService] [elk] loaded module [analysis-common]
[2021-01-12T07:22:16,045][INFO][o.e.p.PluginsService] [elk] loaded module [constant-keyword]
[2021-01-12T07:22:16,046][INFO][o.e.p.PluginsService] [elk] loaded module [flattened]
[2021-01-12T07:22:16,046][INFO][o.e.p.PluginsService] [elk] loaded module [frozen-indices]
[2021-01-12T07:22:16,047][INFO][o.e.p.PluginsService] [elk] loaded module [ingest-common]
[2021-01-12T07:22:16,047][INFO][o.e.p.PluginsService] [elk] loaded module [ingest-geoip]
[2021-01-12T07:22:16,047][INFO][o.e.p.PluginsService] [elk] loaded module [ingest-user-agent]
[2021-01-12T07:22:16,048][INFO][o.e.p.PluginsService] [elk] loaded module [kibana]
...
[2021-01-12T07:22:16,070][INFO][o.e.p.PluginsService] [elk] no plugins loaded
[2021-01-12T07:22:16,399][INFO][o.e.e.NodeEnvironment] [elk] using [1] data paths, mounts [[/ (/dev/mapper/ubuntu--vg-ubuntu--lv)]], net usable_space [55.3gb], net total_space [79.5gb], types [ext4]
[2021-01-12T07:22:16,401][INFO][o.e.e.NodeEnvironment] [elk] heap size [1gb], compressed ordinary object pointers [true]
[2021-01-12T07:22:20,449][INFO][o.e.n.Node] [elk] node name [elk], node ID [hKWW9WC0TAKHr38HOxJIfA], cluster name [elasticsearch], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]

```

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [January 12, 2021, 2:27pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-start-on-vm-boot/260645/4 "2021-01-12T14:27:01Z")

</div>

Looks like it was still starting up when `systemd` decided to time out and kill it. Try a longer timeout.

---

<div class="post-metadata">

**Author:** ![droidus](https://avatars.discourse-cdn.com/v4/letter/d/13edae/32.png) [@droidus](https://discuss.elastic.co/u/droidus)\
**Post date:** [January 12, 2021, 4:32pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-start-on-vm-boot/260645/5 "2021-01-12T16:32:02Z")

</div>

Thank you. That fixed it. I increased from 75 to 6000. Do I also need to set the logstash timeout to prevent it from having no limit? How do I fix this to prevent it from hanging upon shutdown?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [January 12, 2021, 4:37pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-start-on-vm-boot/260645/6 "2021-01-12T16:37:34Z")

</div>

Probably best to ask your Logstash questions on the [Logstash forum](https://discuss.elastic.co/c/elastic-stack/logstash/14), I don't know how to answer them.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 9, 2021, 4:37pm UTC](https://discuss.elastic.co/t/elasticsearch-fails-to-start-on-vm-boot/260645/7 "2021-02-09T16:37:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.