# Unable to start elasticsearch after upgrade 7.17 -\> 8.2 timeout was exceeded

**URL:** <https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960>\
**Category:** Elasticsearch\
**Created:** [July 29, 2022, 8:59am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960 "2022-07-29T08:59:54Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [July 29, 2022, 8:59am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/1 "2022-07-29T08:59:54Z")

</div>

Can you please direct me how to solve the problem when starting elasticsearch node after its upgrade to 8.2

system startup timeouts despite the fact that I adjusted the timeout to 3 minutes  
i followed this procedure

> **[How to prevent systemd service start operation from timing out](https://sleeplessbeastie.eu/2020/02/29/how-to-prevent-systemd-service-start-operation-from-timing-out/)**
>
> Specify systemd startup timeout option to prevent service start operation from timing out.

it seems to be related to geoip databases that are not there, but I do not know how to resolve this

```auto
tode01prahkz:/data/elasticsearch/log# systemctl start elasticsearch
Job for elasticsearch.service failed because a timeout was exceeded. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

```

journalctl

```auto
journalctl -xe -u elasticsearch

-- Unit elasticsearch.service has begun starting up.
Jul 29 10:37:04 tode01prahkz systemd[1]: Got notification message from PID 9976, but reception only permitted for main PID 9909
Jul 29 10:39:49 tode01prahkz systemd[1]: elasticsearch.service start operation timed out. Terminating.
Jul 29 10:39:50 tode01prahkz systemd[1]: Got notification message from PID 9976, but reception only permitted for main PID 9909
Jul 29 10:39:52 tode01prahkz systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Jul 29 10:39:52 tode01prahkz systemd[1]: Unit elasticsearch.service entered failed state.
Jul 29 10:39:52 tode01prahkz systemd[1]: elasticsearch.service failed.

```

the app log:

```auto
...
[2022-07-29T10:37:03,767][ERROR][o.e.i.g.DatabaseNodeService] [tode01prahkz] failed to retrieve database [GeoLite2-Country.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
... 
[2022-07-29T10:37:03,767][ERROR][o.e.i.g.DatabaseNodeService] [tode01prahkz] failed to retrieve database [GeoLite2-ASN.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
...     
[2022-07-29T10:37:03,767][ERROR][o.e.i.g.DatabaseNodeService] [tode01prahkz] failed to retrieve database [GeoLite2-City.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
...        
[2022-07-29T10:37:04,348][INFO][o.e.x.s.a.TokenService] [tode01prahkz] refresh keys
[2022-07-29T10:37:04,463][INFO][o.e.x.s.a.TokenService] [tode01prahkz] refreshed keys
[2022-07-29T10:37:04,503][INFO][o.e.l.LicenseService] [tode01prahkz] license [91cf13b1-6265-4d7f-a516-74b68e5f3f5f] mode [basic] - valid
[2022-07-29T10:37:04,504][INFO][o.e.x.s.a.Realms] [tode01prahkz] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-07-29T10:37:04,509][INFO][o.e.h.AbstractHttpServerTransport] [tode01prahkz] publish_address {10.44.163.37:9200}, bound_addresses {[::]:9200}
[2022-07-29T10:37:04,509][INFO][o.e.n.Node] [tode01prahkz] started {tode01prahkz}{bL_sp4qyS_i1g6NyUA24kA}{667jVFsVReaWGxYnrxdv_g}{tode01prahkz}{10.44.163.37}{10.44.163.37:9300}{hilmst}{ml.max_jvm_size=16106127360, xpack.installed=true, ml.machine_memory=33567277056}
[2022-07-29T10:37:04,873][INFO][o.e.i.g.DatabaseNodeService] [tode01prahkz] successfully loaded geoip database file [GeoLite2-Country.mmdb]
[2022-07-29T10:37:05,001][INFO][o.e.i.g.DatabaseNodeService] [tode01prahkz] successfully loaded geoip database file [GeoLite2-ASN.mmdb]
[2022-07-29T10:37:06,099][INFO][o.e.i.g.DatabaseNodeService] [tode01prahkz] successfully loaded geoip database file [GeoLite2-City.mmdb]
[2022-07-29T10:39:49,639][INFO][o.e.n.Node] [tode01prahkz] stopping ...
[2022-07-29T10:39:49,642][INFO][o.e.x.w.WatcherService] [tode01prahkz] stopping watch service, reason [shutdown initiated]
[2022-07-29T10:39:49,642][INFO][o.e.x.w.WatcherLifeCycleService] [tode01prahkz] watcher has stopped and shutdown
[2022-07-29T10:39:49,953][INFO][o.e.x.m.p.l.CppLogMessageHandler] [tode01prahkz] [controller/10003] [Main.cc@176] ML controller exiting
[2022-07-29T10:39:49,954][INFO][o.e.x.m.p.NativeController] [tode01prahkz] Native controller process has stopped - no new native processes can be started
[2022-07-29T10:39:49,957][INFO][o.e.c.c.Coordinator] [tode01prahkz] master node [{tode03prahkz}{V1Fj3JEaTNyUbS21SkWWgQ}{boDTTy3DT4OOKFLpozIluA}{tode03prahkz}{10.44.163.39}{10.44.163.39:9300}{himst}] disconnected, restarting discovery
[2022-07-29T10:39:50,489][INFO][o.e.n.Node] [tode01prahkz] stopped
[2022-07-29T10:39:50,489][INFO][o.e.n.Node] [tode01prahkz] closing ...
[2022-07-29T10:39:50,499][INFO][o.e.i.g.DatabaseReaderLazyLoader] [tode01prahkz] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-8551826250694294177/geoip-databases/bL_sp4qyS_i1g6NyUA24kA/GeoLite2-Country.mmdb]
[2022-07-29T10:39:50,499][INFO][o.e.i.g.DatabaseReaderLazyLoader] [tode01prahkz] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-8551826250694294177/geoip-databases/bL_sp4qyS_i1g6NyUA24kA/GeoLite2-ASN.mmdb]
[2022-07-29T10:39:50,499][INFO][o.e.i.g.DatabaseReaderLazyLoader] [tode01prahkz] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-8551826250694294177/geoip-databases/bL_sp4qyS_i1g6NyUA24kA/GeoLite2-City.mmdb]
[2022-07-29T10:39:50,500][INFO][o.e.n.Node] [tode01prahkz] closed

```

it hangs on line successfully loaded geoip database file [GeoLite2-City.mmdb]  
which is weird because it complains the file does not exists but later it writes this successfully loaded line and later it timeouts and exits..

the file does not exists here

```auto
tode01prahkz:/usr/share/elasticsearch# find . |grep -i GeoLite2-City.mmdb
tode01prahkz:/usr/share/elasticsearch# ls -l /usr/share/elasticsearch/modules/ingest-geoip
total 2140
-rw-r--r-- 1 root root 56179 Jul 6 17:17 geoip2-3.0.0.jar
-rw-r--r-- 1 root root 100197 Jul 6 17:17 ingest-geoip-8.3.2.jar
-rw-r--r-- 1 root root 75717 Jul 6 17:17 jackson-annotations-2.13.1.jar
-rw-r--r-- 1 root root 374588 Jul 6 17:17 jackson-core-2.13.1.jar
-rw-r--r-- 1 root root 1534017 Jul 6 17:17 jackson-databind-2.13.1.jar
-rw-r--r-- 1 root root 30891 Jul 6 17:17 maxmind-db-2.0.0.jar
-rw-r--r-- 1 root root 1937 Jul 6 17:17 plugin-descriptor.properties
-rw-r--r-- 1 root root 1081 Jul 6 17:17 plugin-security.policy

```

```auto

```

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [July 30, 2022, 9:48am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/2 "2022-07-30T09:48:02Z")

</div>

in order to get rid of geoip err I tried

> [@How to disable geoip usage in 7.14.0](https://discuss.elastic.co/t/how-to-disable-geoip-usage-in-7-14-0/281076/8):
>
> Same for me after Update from 7.13.4. Saw the Log entry after the update start... found ingest.geoip.downloader.enabled: false but with no luck the "false" will be ignored

```auto
PUT _cluster/settings
{
    "persistent": {
       "ingest.geoip.downloader.enabled": false
    }
}

```

but it seems does not help  
the err message dissapeared

there is no ERR message is elasticsearch.log anymore  
the only err I receive is

```auto

Jul 30 11:45:04 tode01prahkz systemd[1]: Got notification message from PID 1147, but reception only permitted for main PID 1080
Jul 30 11:47:49 tode01prahkz systemd[1]: elasticsearch.service start operation timed out. Terminating.
Jul 30 11:47:50 tode01prahkz systemd[1]: Got notification message from PID 1147, but reception only permitted for main PID 1080
Jul 30 11:47:52 tode01prahkz systemd[1]: Failed to start Elasticsearch.

```

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [July 30, 2022, 10:11am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/3 "2022-07-30T10:11:41Z")

</div>

I found out the node joins the clster but it is shutted down later  
the problem is this err

```auto

Jul 30 12:05:17 tode01prahkz systemd[1]: Starting Elasticsearch...
Jul 30 12:05:33 tode01prahkz systemd[1]: Got notification message from PID 3780, but reception only permitted for main PID 3714
Jul 30 12:08:18 tode01prahkz systemd[1]: elasticsearch.service start operation timed out. Terminating.
Jul 30 12:08:19 tode01prahkz systemd[1]: Got notification message from PID 3780, but reception only permitted for main PID 3714
Jul 30 12:08:21 tode01prahkz systemd[1]: Failed to start Elasticsearch.
Jul 30 12:08:21 tode01prahkz systemd[1]: Unit elasticsearch.service entered failed state.
Jul 30 12:08:21 tode01prahkz systemd[1]: elasticsearch.service failed.

```

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [July 30, 2022, 12:51pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/4 "2022-07-30T12:51:05Z")

</div>

the issue is here with JAVA  
see err complain of pid 23374 and 23308

```auto
tode01prahkz:/data/elasticsearch/log# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─startup-timeout.conf
   Active: activating (start) since Sat 2022-07-30 14:48:26 CEST; 25s ago
     Docs: https://www.elastic.co
 Main PID: 23308 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─23308 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.libs=lib/tools/server-...
           ├─23374 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouch -Xss1m -D...
           └─23400 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Jul 30 14:48:26 tode01prahkz systemd[1]: Starting Elasticsearch...
Jul 30 14:48:41 tode01prahkz systemd[1]: Got notification message from PID 23374, but reception only permitted for main PID 23308

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 1, 2022, 10:35pm UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/5 "2022-08-01T22:35:17Z")

</div>

> [@Petr.Simik](#):
>
> ```auto
> [2022-07-29T10:39:49,639][INFO][o.e.n.Node] [tode01prahkz] stopping ...
> 
> ```

This shows that something asked the Elasticsearch process to shutdown.  
Maybe there's a systemd timeout you can increase to let Elasticsearch start up?

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [August 2, 2022, 6:15am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/6 "2022-08-02T06:15:55Z")

</div>

thank you @warkolm I have upated the timeout to 3minutes  
by looking at java CGroup there are two java processes

> CGroup: /system.slice/elasticsearch.service  
> ├─23308 /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+U  
> ├─23374 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.c  
> └─23400 /usr/share/elasticsearch/modules/x-pack-ml/platform/

but in 7.17 version this looked

> tode01prahkz:~# systemctl status elasticsearch  
> ...  
> Main PID: 13760 (java)  
> CGroup: /system.slice/elasticsearch.service  
> ├─13760 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.n...  
> └─13965 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux

I have downgraded the elastic back to 7.17 and I will try it again.

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [August 2, 2022, 6:40am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/7 "2022-08-02T06:40:34Z")

</div>

@warkolm pls do you know whether there were any changes within JAVA setup I should process before moving from v7 to v8

when I run the upgrade process again I am getting the same err message

it starts the cluster but after timeout the cluster stops

> tode01prahkz:~# systemctl status elasticsearch  
> ● elasticsearch.service - Elasticsearch  
> Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)  
> Drop-In: /etc/systemd/system/elasticsearch.service.d  
> └─startup-timeout.conf  
> Active: activating (start) since Tue 2022-08-02 08:35:13 CEST; 1min 13s ago  
> Docs: [https://www.elastic.co](https://www.elastic.co)  
> Main PID: 22212 (java)  
> CGroup: /system.slice/elasticsearch.service  
> ├─ **22212** /usr/share/elasticsearch/jdk/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=/usr/share/elasticsearch/bin/elasticsearch -Dcli.libs=lib/tools/serve...  
> ├─ **22284** /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=allow -XX:+AlwaysPreTouch -Xss1m ...  
> └─22309 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86\_64/bin/controller
> 
> Aug 02 08:35:13 tode01prahkz systemd[1]: Starting Elasticsearch...  
> Aug 02 08:35:31 tode01prahkz systemd[1]: Got notification message from PID **22284** , but reception only permitted for main PID **22212**

this is how it looks after timeout

```auto
> tode01prahkz:~# systemctl status elasticsearch
> ● elasticsearch.service - Elasticsearch
> Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
> Drop-In: /etc/systemd/system/elasticsearch.service.d
> └─startup-timeout.conf
> Active: failed (Result: timeout) since Tue 2022-08-02 08:38:16 CEST; 3min 49s ago
> Docs: https://www.elastic.co
> Process: 22212 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=143)
> Main PID: 22212 (code=exited, status=143)
> 
> Aug 02 08:35:13 tode01prahkz systemd[1]: Starting Elasticsearch...
> Aug 02 08:35:31 tode01prahkz systemd[1]: Got notification message from PID 22284, but reception only permitted for main PID 22212
> Aug 02 08:38:13 tode01prahkz systemd[1]: elasticsearch.service start operation timed out. Terminating.
> Aug 02 08:38:14 tode01prahkz systemd[1]: Got notification message from PID 22284, but reception only permitted for main PID 22212
> Aug 02 08:38:16 tode01prahkz systemd[1]: Failed to start Elasticsearch.
> Aug 02 08:38:16 tode01prahkz systemd[1]: Unit elasticsearch.service entered failed state.
> Aug 02 08:38:16 tode01prahkz systemd[1]: elasticsearch.service failed.

```

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [August 2, 2022, 6:53am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/8 "2022-08-02T06:53:52Z")

</div>

after reinstall It shows different err in elastic.log which may be the root cause  
I still do not know how to resolve it but I am gooing to search this issue

```auto
> ...
> [2022-08-02T08:49:22,597][INFO][o.e.x.s.a.RealmsAuthenticator] [tode01prahkz] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
> [2022-08-02T08:49:22,597][INFO][o.e.x.s.a.RealmsAuthenticator] [tode01prahkz] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
> [2022-08-02T08:49:22,598][INFO][o.e.x.s.a.RealmsAuthenticator] [tode01prahkz] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
> [2022-08-02T08:49:22,597][INFO][o.e.x.s.a.RealmsAuthenticator] [tode01prahkz] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
> [2022-08-02T08:49:22,669][INFO][o.e.x.s.a.RealmsAuthenticator] [tode01prahkz] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
> [2022-08-02T08:49:29,462][INFO][o.e.n.Node] [tode01prahkz] stopping ...
> [2022-08-02T08:49:29,463][INFO][o.e.x.w.WatcherService] [tode01prahkz] stopping watch service, reason [shutdown initiated]
> [2022-08-02T08:49:29,464][INFO][o.e.x.w.WatcherLifeCycleService] [tode01prahkz] watcher has stopped and shutdown
> [2022-08-02T08:49:29,750][INFO][o.e.x.m.p.l.CppLogMessageHandler] [tode01prahkz] [controller/23860] [Main.cc@176] ML controller exiting
> [2022-08-02T08:49:29,750][INFO][o.e.x.m.p.NativeController] [tode01prahkz] Native controller process has stopped - no new native processes can be started
> [2022-08-02T08:49:29,752][INFO][o.e.c.c.Coordinator] [tode01prahkz] master node [{tode03prahkz}{V1Fj3JEaTNyUbS21SkWWgQ}{boDTTy3DT4OOKFLpozIluA}{tode03prahkz}{10.44.163.39}{10.44.163.39:9300}{himst}] disconnected, restarting discovery
> [2022-08-02T08:49:29,807][INFO][o.e.n.Node] [tode01prahkz] stopped
> [2022-08-02T08:49:29,807][INFO][o.e.n.Node] [tode01prahkz] closing ...
> [2022-08-02T08:49:29,815][INFO][o.e.n.Node] [tode01prahkz] closed

```

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [August 2, 2022, 7:22am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/9 "2022-08-02T07:22:27Z")

</div>

Do you know why the node complains of realsm auth problem ?

```auto
[2022-08-02T09:16:20,407][INFO][o.e.n.Node] [tode01prahkz] starting ...
[2022-08-02T09:16:20,432][INFO][o.e.x.s.c.f.PersistentCache] [tode01prahkz] persistent cache index loaded
[2022-08-02T09:16:20,433][INFO][o.e.x.d.l.DeprecationIndexingComponent] [tode01prahkz] deprecation component started
[2022-08-02T09:16:20,516][INFO][o.e.t.TransportService] [tode01prahkz] publish_address {10.44.163.37:9300}, bound_addresses {[::]:9300}
[2022-08-02T09:16:21,940][INFO][o.e.b.BootstrapChecks] [tode01prahkz] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-08-02T09:16:21,955][WARN][o.e.c.c.ClusterBootstrapService] [tode01prahkz] this node is locked into cluster UUID [XtdLLc-TTrCiKcJ5WhtFLw] but [cluster.initial_master_nodes] is set to [tode01prahkz, tode02prahkz, tode03prahkz]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts
[2022-08-02T09:16:23,133][INFO][o.e.c.s.ClusterApplierService] [tode01prahkz] master node changed {previous [], current [{tode03prahkz}{V1Fj3JEaTNyUbS21SkWWgQ}{boDTTy3DT4OOKFLpozIluA}{tode03prahkz}{10.44.163.39}{10.44.163.39:9300}{himst}]}, added {{todk01prahkz}{fcb4e-ArSKeozVMFaTfAew}{pKVSzPs0QXmgDe3kYsqVOg}{todk01prahkz}{10.44.163.40}{10.44.163.40:9300}, {tode03prahkz}{V1Fj3JEaTNyUbS21SkWWgQ}{boDTTy3DT4OOKFLpozIluA}{tode03prahkz}{10.44.163.39}{10.44.163.39:9300}{himst}, {tode02prahkz}{HY5GXbZvRVSczTsrEZG_DQ}{9sHZi7hiTXOGdZMHbkONZw}{tode02prahkz}{10.44.163.38}{10.44.163.38:9300}{himst}}, term: 22, version: 525973, reason: ApplyCommitRequest{term=22, version=525973, sourceNode={tode03prahkz}{V1Fj3JEaTNyUbS21SkWWgQ}{boDTTy3DT4OOKFLpozIluA}{tode03prahkz}{10.44.163.39}{10.44.163.39:9300}{himst}{xpack.installed=true, data=hot, transform.node=true}}
[2022-08-02T09:16:23,240][INFO][o.e.c.s.ClusterSettings] [tode01prahkz] updating [cluster.routing.allocation.enable] from [all] to [primaries]
[2022-08-02T09:16:23,241][INFO][o.e.c.s.ClusterSettings] [tode01prahkz] updating [cluster.routing.allocation.node_concurrent_incoming_recoveries] from [2] to [5]
[2022-08-02T09:16:23,241][INFO][o.e.c.s.ClusterSettings] [tode01prahkz] updating [cluster.routing.allocation.node_concurrent_outgoing_recoveries] from [2] to [5]
[2022-08-02T09:16:23,241][INFO][o.e.c.s.ClusterSettings] [tode01prahkz] updating [cluster.routing.allocation.enable] from [all] to [primaries]
[2022-08-02T09:16:23,241][INFO][o.e.c.s.ClusterSettings] [tode01prahkz] updating [ingest.geoip.downloader.enabled] from [true] to [false]
[2022-08-02T09:16:23,242][INFO][o.e.c.s.ClusterSettings] [tode01prahkz] updating [xpack.monitoring.elasticsearch.collection.enabled] from [true] to [false]
[2022-08-02T09:16:23,242][INFO][o.e.c.s.ClusterSettings] [tode01prahkz] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2022-08-02T09:16:23,842][INFO][o.e.x.s.a.TokenService] [tode01prahkz] refresh keys
[2022-08-02T09:16:23,956][INFO][o.e.x.s.a.TokenService] [tode01prahkz] refreshed keys
[2022-08-02T09:16:23,985][INFO][o.e.l.LicenseService] [tode01prahkz] license [91cf13b1-6265-4d7f-a516-74b68e5f3f5f] mode [basic] - valid
[2022-08-02T09:16:23,986][INFO][o.e.x.s.a.Realms] [tode01prahkz] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-08-02T09:16:23,991][INFO][o.e.h.AbstractHttpServerTransport] [tode01prahkz] publish_address {10.44.163.37:9200}, bound_addresses {[::]:9200}
[2022-08-02T09:16:23,991][INFO][o.e.n.Node] [tode01prahkz] started {tode01prahkz}{aCiSpclBTmGc1gvS7TImoA}{l9ewzpBsT-SF75UfzwlCkg}{tode01prahkz}{10.44.163.37}{10.44.163.37:9300}{hilmst}{ml.max_jvm_size=16106127360, xpack.installed=true, ml.machine_memory=33567244288}
[2022-08-02T09:16:32,607][INFO][o.e.x.s.a.RealmsAuthenticator] [tode01prahkz] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2022-08-02T09:16:32,607][INFO][o.e.x.s.a.RealmsAuthenticator] [tode01prahkz] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2022-08-02T09:16:32,607][INFO][o.e.x.s.a.RealmsAuthenticator] [tode01prahkz] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2022-08-02T09:16:32,607][INFO][o.e.x.s.a.RealmsAuthenticator] [tode01prahkz] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2022-08-02T09:16:32,608][INFO][o.e.x.s.a.RealmsAuthenticator] [tode01prahkz] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]

```

before timeout it is normally connected to the cluster and I can access it with password

```auto
xxxxx@tode01prahkz:~$ curl -u elastic:xxx "http://localhost:9200/_cat/nodes?v&h=name,version,node.role,master"
name version node.role master
tode03prahkz 7.17.0 himst *
tode02prahkz 7.17.0 himst -
todk01prahkz 7.17.0 - -
tode01prahkz 8.3.2 hilmst -

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 2, 2022, 8:11am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/10 "2022-08-02T08:11:36Z")

</div>

> [@Petr.Simik](#):
>
> Do you know why the node complains of realsm auth problem ?

Because something is trying to login with the `elastic` user with a wrong password.

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [August 2, 2022, 11:10am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/11 "2022-08-02T11:10:07Z")

</div>

@warkolm  
password is not stored in elasticsearch.yml nor in other conf files and nodes are joined based on keys but not elastic password.

i did just: systemctl start elasticsearch

Where do you think the incorrect login is comming from?

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [August 3, 2022, 5:11am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/12 "2022-08-03T05:11:26Z")

</div>

false alarm: this auth err was caused by metricbeat

> Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]

I forgot to fix this conf.

however the original issue still remains unresolved

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [August 3, 2022, 5:59am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/13 "2022-08-03T05:59:56Z")

</div>

It seems I have resolved it by adding **NotifyAccess=all**  
to service.d conf file

since then it starts normally

> tode01prahkz:/etc/systemd/system/elasticsearch.service.d# cat /etc/systemd/system/elasticsearch.service.d/startup-timeout.conf  
> [Service]  
> TimeoutStartSec=180  
> NotifyAccess=all

---

<div class="post-metadata">

**Author:** ![Petr.Simik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/petr.simik/32/38082_2.png) [@Petr.Simik](https://discuss.elastic.co/u/Petr.Simik)\
**Post date:** [August 3, 2022, 6:02am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/14 "2022-08-03T06:02:40Z")

</div>

it is related to

> [@Update errors to version 8.3.1](https://discuss.elastic.co/t/update-errors-to-version-8-3-1/308808/2):
>
> Hello, As I don't have too much feedback on this problem, I will bring the feedback on the command : journalctl -u elasticsearch.service And the last start of the service "Jul 05 15:56:30" : juil. 05 15:56:30 STL-SAS-025 systemd[1]: Starting Elasticsearch... juil. 05 15:56:33 STL-SAS-025 systemd-entrypoint[32484]: [2022-07-05T15:56:33,669][INFO][o.e.n.Node] [Elasticsearch-Logs-1] version[8.3.1], \> juil. 05 15:56:33 STL-SAS-025 systemd-entrypoint[32484]: [2022-07-05T15:56:33,…

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 31, 2022, 6:03am UTC](https://discuss.elastic.co/t/unable-to-start-elasticsearch-after-upgrade-7-17-8-2-timeout-was-exceeded/310960/15 "2022-08-31T06:03:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.