Can you please direct me how to solve the problem when starting elasticsearch node after its upgrade to 8.2
system startup timeouts despite the fact that I adjusted the timeout to 3 minutes
i followed this procedure
it seems to be related to geoip databases that are not there, but I do not know how to resolve this
tode01prahkz:/data/elasticsearch/log# systemctl start elasticsearch
Job for elasticsearch.service failed because a timeout was exceeded. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
journalctl
journalctl -xe -u elasticsearch
-- Unit elasticsearch.service has begun starting up.
Jul 29 10:37:04 tode01prahkz systemd[1]: Got notification message from PID 9976, but reception only permitted for main PID 9909
Jul 29 10:39:49 tode01prahkz systemd[1]: elasticsearch.service start operation timed out. Terminating.
Jul 29 10:39:50 tode01prahkz systemd[1]: Got notification message from PID 9976, but reception only permitted for main PID 9909
Jul 29 10:39:52 tode01prahkz systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Jul 29 10:39:52 tode01prahkz systemd[1]: Unit elasticsearch.service entered failed state.
Jul 29 10:39:52 tode01prahkz systemd[1]: elasticsearch.service failed.
the app log:
...
[2022-07-29T10:37:03,767][ERROR][o.e.i.g.DatabaseNodeService] [tode01prahkz] failed to retrieve database [GeoLite2-Country.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
...
[2022-07-29T10:37:03,767][ERROR][o.e.i.g.DatabaseNodeService] [tode01prahkz] failed to retrieve database [GeoLite2-ASN.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
...
[2022-07-29T10:37:03,767][ERROR][o.e.i.g.DatabaseNodeService] [tode01prahkz] failed to retrieve database [GeoLite2-City.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
...
[2022-07-29T10:37:04,348][INFO ][o.e.x.s.a.TokenService ] [tode01prahkz] refresh keys
[2022-07-29T10:37:04,463][INFO ][o.e.x.s.a.TokenService ] [tode01prahkz] refreshed keys
[2022-07-29T10:37:04,503][INFO ][o.e.l.LicenseService ] [tode01prahkz] license [91cf13b1-6265-4d7f-a516-74b68e5f3f5f] mode [basic] - valid
[2022-07-29T10:37:04,504][INFO ][o.e.x.s.a.Realms ] [tode01prahkz] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2022-07-29T10:37:04,509][INFO ][o.e.h.AbstractHttpServerTransport] [tode01prahkz] publish_address {10.44.163.37:9200}, bound_addresses {[::]:9200}
[2022-07-29T10:37:04,509][INFO ][o.e.n.Node ] [tode01prahkz] started {tode01prahkz}{bL_sp4qyS_i1g6NyUA24kA}{667jVFsVReaWGxYnrxdv_g}{tode01prahkz}{10.44.163.37}{10.44.163.37:9300}{hilmst}{ml.max_jvm_size=16106127360, xpack.installed=true, ml.machine_memory=33567277056}
[2022-07-29T10:37:04,873][INFO ][o.e.i.g.DatabaseNodeService] [tode01prahkz] successfully loaded geoip database file [GeoLite2-Country.mmdb]
[2022-07-29T10:37:05,001][INFO ][o.e.i.g.DatabaseNodeService] [tode01prahkz] successfully loaded geoip database file [GeoLite2-ASN.mmdb]
[2022-07-29T10:37:06,099][INFO ][o.e.i.g.DatabaseNodeService] [tode01prahkz] successfully loaded geoip database file [GeoLite2-City.mmdb]
[2022-07-29T10:39:49,639][INFO ][o.e.n.Node ] [tode01prahkz] stopping ...
[2022-07-29T10:39:49,642][INFO ][o.e.x.w.WatcherService ] [tode01prahkz] stopping watch service, reason [shutdown initiated]
[2022-07-29T10:39:49,642][INFO ][o.e.x.w.WatcherLifeCycleService] [tode01prahkz] watcher has stopped and shutdown
[2022-07-29T10:39:49,953][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [tode01prahkz] [controller/10003] [Main.cc@176] ML controller exiting
[2022-07-29T10:39:49,954][INFO ][o.e.x.m.p.NativeController] [tode01prahkz] Native controller process has stopped - no new native processes can be started
[2022-07-29T10:39:49,957][INFO ][o.e.c.c.Coordinator ] [tode01prahkz] master node [{tode03prahkz}{V1Fj3JEaTNyUbS21SkWWgQ}{boDTTy3DT4OOKFLpozIluA}{tode03prahkz}{10.44.163.39}{10.44.163.39:9300}{himst}] disconnected, restarting discovery
[2022-07-29T10:39:50,489][INFO ][o.e.n.Node ] [tode01prahkz] stopped
[2022-07-29T10:39:50,489][INFO ][o.e.n.Node ] [tode01prahkz] closing ...
[2022-07-29T10:39:50,499][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [tode01prahkz] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-8551826250694294177/geoip-databases/bL_sp4qyS_i1g6NyUA24kA/GeoLite2-Country.mmdb]
[2022-07-29T10:39:50,499][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [tode01prahkz] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-8551826250694294177/geoip-databases/bL_sp4qyS_i1g6NyUA24kA/GeoLite2-ASN.mmdb]
[2022-07-29T10:39:50,499][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [tode01prahkz] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-8551826250694294177/geoip-databases/bL_sp4qyS_i1g6NyUA24kA/GeoLite2-City.mmdb]
[2022-07-29T10:39:50,500][INFO ][o.e.n.Node ] [tode01prahkz] closed
it hangs on line successfully loaded geoip database file [GeoLite2-City.mmdb]
which is weird because it complains the file does not exists but later it writes this successfully loaded line and later it timeouts and exits..
the file does not exists here
tode01prahkz:/usr/share/elasticsearch# find . |grep -i GeoLite2-City.mmdb
tode01prahkz:/usr/share/elasticsearch# ls -l /usr/share/elasticsearch/modules/ingest-geoip
total 2140
-rw-r--r-- 1 root root 56179 Jul 6 17:17 geoip2-3.0.0.jar
-rw-r--r-- 1 root root 100197 Jul 6 17:17 ingest-geoip-8.3.2.jar
-rw-r--r-- 1 root root 75717 Jul 6 17:17 jackson-annotations-2.13.1.jar
-rw-r--r-- 1 root root 374588 Jul 6 17:17 jackson-core-2.13.1.jar
-rw-r--r-- 1 root root 1534017 Jul 6 17:17 jackson-databind-2.13.1.jar
-rw-r--r-- 1 root root 30891 Jul 6 17:17 maxmind-db-2.0.0.jar
-rw-r--r-- 1 root root 1937 Jul 6 17:17 plugin-descriptor.properties
-rw-r--r-- 1 root root 1081 Jul 6 17:17 plugin-security.policy