below is an log before a restart - then after a restart
[root@prod-resultselasticsearchm-101w logs]# cat prod-resultselasticsearch.log
[2022-09-16T20:15:28,242][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb.tmp.gz]
[2022-09-16T20:15:28,760][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [prod-resultselasticsearchm-101w] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb]
[2022-09-16T20:15:28,760][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] successfully reloaded changed geoip database file [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb]
[2022-09-16T20:15:32,049][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb.tmp.gz]
[2022-09-16T20:15:33,593][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb.tmp.gz]
[2022-09-16T20:15:33,738][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [prod-resultselasticsearchm-101w] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb]
[2022-09-16T20:15:33,738][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] successfully reloaded changed geoip database file [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb]
[2022-09-16T20:15:33,998][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [prod-resultselasticsearchm-101w] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb]
[2022-09-16T20:15:33,999][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] successfully reloaded changed geoip database file [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb]
[root@prod-resultselasticsearchm-101w logs]# systemctl restart elasticsearch
[root@prod-resultselasticsearchm-101w logs]# cat prod-resultselasticsearch.log
[2022-09-19T13:10:47,433][INFO ][o.e.n.Node ] [prod-resultselasticsearchm-101w] stopping ...
[2022-09-19T13:10:47,449][INFO ][o.e.x.w.WatcherService ] [prod-resultselasticsearchm-101w] stopping watch service, reason [shutdown initiated]
[2022-09-19T13:10:47,450][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [prod-resultselasticsearchm-101w] [controller/61242] [Main.cc@174] ML controller exiting
[2022-09-19T13:10:47,451][INFO ][o.e.x.m.p.NativeController] [prod-resultselasticsearchm-101w] Native controller process has stopped - no new native processes can be started
[2022-09-19T13:10:47,452][INFO ][o.e.x.w.WatcherLifeCycleService] [prod-resultselasticsearchm-101w] watcher has stopped and shutdown
[2022-09-19T13:10:47,453][INFO ][o.e.c.c.Coordinator ] [prod-resultselasticsearchm-101w] master node [{prod-resultselasticsearchm-103w}{Q0xgQdbEQJGEAEFKM6On3w}{rXBiNTcJQ7qQo7mRPc3cjQ}{10.212.39.93}{10.212.39.93:9300}{ilmr}] disconnected, restarting discovery
[2022-09-19T13:10:47,473][INFO ][o.e.n.Node ] [prod-resultselasticsearchm-101w] stopped
[2022-09-19T13:10:47,473][INFO ][o.e.n.Node ] [prod-resultselasticsearchm-101w] closing ...
[2022-09-19T13:10:47,483][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [prod-resultselasticsearchm-101w] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb]
[2022-09-19T13:10:47,483][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [prod-resultselasticsearchm-101w] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb]
[2022-09-19T13:10:47,483][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [prod-resultselasticsearchm-101w] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-3165155470998174077/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb]
[2022-09-19T13:10:47,484][INFO ][o.e.n.Node ] [prod-resultselasticsearchm-101w] closed
[2022-09-19T13:10:50,272][INFO ][o.e.n.Node ] [prod-resultselasticsearchm-101w] version[7.17.5], pid[106803], build[default/rpm/8d61b4f7ddf931f219e3745f295ed2bbc50c8e84/2022-06-23T21:57:28.736740635Z], OS[Linux/3.10.0-1160.71.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/18.0.1.1/18.0.1.1+2-6]
[2022-09-19T13:10:50,273][INFO ][o.e.n.Node ] [prod-resultselasticsearchm-101w] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2022-09-19T13:10:50,273][INFO ][o.e.n.Node ] [prod-resultselasticsearchm-101w] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Djava.security.manager=allow, -Xms4g, -Xmx4g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-4947918882176097480, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/elasticsearch/logs, -XX:ErrorFile=/elasticsearch/logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/elasticsearch/logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=2147483648, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2022-09-19T13:10:52,167][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [aggs-matrix-stats]
[2022-09-19T13:10:52,167][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [analysis-common]
[2022-09-19T13:10:52,167][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [constant-keyword]
[2022-09-19T13:10:52,168][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [frozen-indices]
[2022-09-19T13:10:52,168][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [ingest-common]
[2022-09-19T13:10:52,168][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [ingest-geoip]
[2022-09-19T13:10:52,168][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [ingest-user-agent]
[2022-09-19T13:10:52,168][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [kibana]
[2022-09-19T13:10:52,168][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [lang-expression]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [lang-mustache]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [lang-painless]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [legacy-geo]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [mapper-extras]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [mapper-version]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [parent-join]
[2022-09-19T13:10:52,169][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [percolator]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [rank-eval]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [reindex]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [repositories-metering-api]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [repository-encrypted]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [repository-url]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [runtime-fields-common]
[2022-09-19T13:10:52,170][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [search-business-rules]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [searchable-snapshots]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [snapshot-repo-test-kit]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [spatial]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [systemd]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [transform]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [transport-netty4]
[2022-09-19T13:10:52,171][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [unsigned-long]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [vector-tile]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [vectors]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [wildcard]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-aggregate-metric]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-analytics]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-async]
[2022-09-19T13:10:52,172][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-async-search]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-autoscaling]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-ccr]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-core]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-data-streams]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-deprecation]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-enrich]
[2022-09-19T13:10:52,173][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-eql]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-fleet]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-graph]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-identity-provider]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-ilm]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-logstash]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-ml]
[2022-09-19T13:10:52,174][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-monitoring]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-ql]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-rollup]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-security]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-shutdown]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-sql]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-stack]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-text-structure]
[2022-09-19T13:10:52,175][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-voting-only-node]
[2022-09-19T13:10:52,176][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] loaded module [x-pack-watcher]
[2022-09-19T13:10:52,176][INFO ][o.e.p.PluginsService ] [prod-resultselasticsearchm-101w] no plugins loaded
[2022-09-19T13:10:52,210][INFO ][o.e.e.NodeEnvironment ] [prod-resultselasticsearchm-101w] using [1] data paths, mounts [[/elasticsearch (/dev/mapper/secret)]], net usable_space [9.1gb], net total_space [9.7gb], types [ext4]
[2022-09-19T13:10:52,210][INFO ][o.e.e.NodeEnvironment ] [prod-resultselasticsearchm-101w] heap size [4gb], compressed ordinary object pointers [true]
[2022-09-19T13:10:52,261][INFO ][o.e.n.Node ] [prod-resultselasticsearchm-101w] node name [prod-resultselasticsearchm-101w], node ID [ZN56Y0twTkaQOVlD86bPDQ], cluster name [prod-resultselasticsearch], roles [master, remote_cluster_client, ml, ingest]
[2022-09-19T13:10:56,573][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [prod-resultselasticsearchm-101w] [controller/107002] [Main.cc@122] controller (64 bit): Version 7.17.5 (Build 6070d498c77671) Copyright (c) 2022 Elasticsearch BV
[2022-09-19T13:10:56,882][INFO ][o.e.i.g.ConfigDatabases ] [prod-resultselasticsearchm-101w] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes
[2022-09-19T13:10:56,883][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ]
[2022-09-19T13:10:57,363][INFO ][o.e.t.NettyAllocator ] [prod-resultselasticsearchm-101w] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2022-09-19T13:10:57,386][INFO ][o.e.i.r.RecoverySettings ] [prod-resultselasticsearchm-101w] using rate limit [15mb] with [default=15mb, read=0b, write=0b, max=0b]
[2022-09-19T13:10:57,417][INFO ][o.e.d.DiscoveryModule ] [prod-resultselasticsearchm-101w] using discovery type [zen] and seed hosts providers [settings]
[2022-09-19T13:10:57,797][INFO ][o.e.g.DanglingIndicesState] [prod-resultselasticsearchm-101w] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-09-19T13:10:58,351][INFO ][o.e.n.Node ] [prod-resultselasticsearchm-101w] initialized
[2022-09-19T13:10:58,351][INFO ][o.e.n.Node ] [prod-resultselasticsearchm-101w] starting ...
[2022-09-19T13:10:58,354][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [prod-resultselasticsearchm-101w] deprecation component started
[2022-09-19T13:10:58,433][INFO ][o.e.t.TransportService ] [prod-resultselasticsearchm-101w] publish_address {10.212.39.91:9300}, bound_addresses {10.212.39.91:9300}
[2022-09-19T13:10:58,711][INFO ][o.e.b.BootstrapChecks ] [prod-resultselasticsearchm-101w] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-09-19T13:10:58,713][INFO ][o.e.c.c.Coordinator ] [prod-resultselasticsearchm-101w] cluster UUID [ztoDRCgNTome-NHt6sZk1g]
[2022-09-19T13:10:58,939][INFO ][o.e.c.s.ClusterApplierService] [prod-resultselasticsearchm-101w] master node changed {previous [], current [{prod-resultselasticsearchm-103w}{Q0xgQdbEQJGEAEFKM6On3w}{rXBiNTcJQ7qQo7mRPc3cjQ}{10.212.39.93}{10.212.39.93:9300}{ilmr}]}, added {{prod-resultselasticsearchd-105w}{Iij5WOV8QIqHVcRL-snQPQ}{GsUqbklKTJOzTEaoLbdOIw}{10.212.39.100}{10.212.39.100:9300}{cdfhilrstw}, {prod-resultselasticsearchd-102w}{osq8RkuFSWyqaUTaejwVFQ}{znmszTK9TnCUjkNmn5YirQ}{10.212.39.97}{10.212.39.97:9300}{cdfhilrstw}, {prod-resultselasticsearchm-103w}{Q0xgQdbEQJGEAEFKM6On3w}{rXBiNTcJQ7qQo7mRPc3cjQ}{10.212.39.93}{10.212.39.93:9300}{ilmr}, {prod-resultselasticsearchd-101w}{3BDuEpdkRJKggzrWdS9zMA}{pllY9XMqRcOR24LIxwyVwg}{10.212.39.96}{10.212.39.96:9300}{cdfhilrstw}, {prod-resultselasticsearchm-102w}{LjJlMVMJSRWgeyKtyINfcg}{V5AFlcLdR0CjDh189zAwKA}{10.212.39.92}{10.212.39.92:9300}{ilmr}, {prod-resultselasticsearchm-105w}{pL2YO8XyRgq96fZ6YE3qgg}{KtLZs7PwR9KUnBOmmVOv8g}{10.212.39.95}{10.212.39.95:9300}{ilmr}, {prod-resultselasticsearchd-110w}{r5xpKmoiQQKe0cyS2ShHxw}{mNEdt9z7RCWzCzu6C7hPcA}{10.212.39.105}{10.212.39.105:9300}{cdfhilrstw}, {prod-resultselasticsearchd-109w}{i8WCq64mQ4-nrJX0r4AHGg}{XUL7eCYqR7C6hMd6pfUhHA}{10.212.39.104}{10.212.39.104:9300}{cdfhilrstw}, {prod-resultselasticsearchd-107w}{NnIiBZB0RZeIkEDDSYgjCw}{CcvKLdkJSY2bdHvzW1awqA}{10.212.39.102}{10.212.39.102:9300}{cdfhilrstw}, {prod-resultselasticsearchd-106w}{HDttVhUiSySauKb6rilrMA}{1MwzqAeYSw6iiplirXYYvQ}{10.212.39.101}{10.212.39.101:9300}{cdfhilrstw}, {prod-resultselasticsearchc-102w}{L8dAHMApQb-UnfgsOTdk3A}{nc_szeNJR1eum4tfWqpvQw}{10.212.39.107}{10.212.39.107:9300}{ilr}, {prod-resultselasticsearchc-103w}{qS8r9qsBTGCdztJD0GoQLQ}{21nq-DZKRZepjT350XQ75Q}{10.212.39.108}{10.212.39.108:9300}{ilr}, {prod-resultselasticsearchd-103w}{6kYumoEZTsuPukfNnscaqw}{-mNyesuKT9aAQ9htXzEiDA}{10.212.39.98}{10.212.39.98:9300}{cdfhilrstw}, {prod-resultselasticsearchd-108w}{YzwOTkz1RLeSNPBNZdb4kg}{y4zFYTBgT7CUMXEQlZWvsQ}{10.212.39.103}{10.212.39.103:9300}{cdfhilrstw}, {prod-resultselasticsearchd-104w}{z2sXRtOQQUOWoMS6rj18Nw}{8VNNjZz4RUiLaxC8VL6N7A}{10.212.39.99}{10.212.39.99:9300}{cdfhilrstw}, {prod-resultselasticsearchc-101w}{DE8r-HjwQmWxWy2gsD2F3A}{QqqzL_E1Sq2ij-tLI40PJw}{10.212.39.106}{10.212.39.106:9300}{ilr}, {prod-resultselasticsearchm-104w}{H1656CU4R5S8SbiKQcyxuA}{8MgjKpCbSsuDyWy3lN0OKw}{10.212.39.94}{10.212.39.94:9300}{ilmr}}, term: 80, version: 1338, reason: ApplyCommitRequest{term=80, version=1338, sourceNode={prod-resultselasticsearchm-103w}{Q0xgQdbEQJGEAEFKM6On3w}{rXBiNTcJQ7qQo7mRPc3cjQ}{10.212.39.93}{10.212.39.93:9300}{ilmr}{ml.machine_memory=8181776384, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=4294967296, transform.node=false}}
[2022-09-19T13:10:58,992][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb.tmp.gz]
[2022-09-19T13:10:58,993][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb.tmp.gz]
[2022-09-19T13:10:58,994][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb.tmp.gz]
[2022-09-19T13:10:58,999][ERROR][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] failed to retrieve database [GeoLite2-Country.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:929) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:285) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:51) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:542) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.ingest.geoip.DatabaseNodeService.lambda$retrieveDatabase$11(DatabaseNodeService.java:376) [ingest-geoip-7.17.5.jar:7.17.5]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.5.jar:7.17.5]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-09-19T13:10:59,006][ERROR][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] failed to retrieve database [GeoLite2-City.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:929) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:285) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:51) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:542) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.ingest.geoip.DatabaseNodeService.lambda$retrieveDatabase$11(DatabaseNodeService.java:376) [ingest-geoip-7.17.5.jar:7.17.5]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.5.jar:7.17.5]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-09-19T13:10:59,007][ERROR][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] failed to retrieve database [GeoLite2-ASN.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:929) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:285) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:51) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:542) ~[elasticsearch-7.17.5.jar:7.17.5]
at org.elasticsearch.ingest.geoip.DatabaseNodeService.lambda$retrieveDatabase$11(DatabaseNodeService.java:376) [ingest-geoip-7.17.5.jar:7.17.5]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.5.jar:7.17.5]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-09-19T13:10:59,201][INFO ][o.e.l.LicenseService ] [prod-resultselasticsearchm-101w] license [f069c374-fbe5-436c-ad8d-f8ebd6d61fe9] mode [basic] - valid
[2022-09-19T13:10:59,212][INFO ][o.e.h.AbstractHttpServerTransport] [prod-resultselasticsearchm-101w] publish_address {10.212.39.91:9200}, bound_addresses {10.212.39.91:9200}
[2022-09-19T13:10:59,212][INFO ][o.e.n.Node ] [prod-resultselasticsearchm-101w] started
[2022-09-19T13:10:59,993][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb.tmp.gz]
[2022-09-19T13:10:59,994][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb.tmp.gz]
[2022-09-19T13:10:59,994][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb.tmp.gz]
[2022-09-19T13:11:00,244][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] successfully reloaded changed geoip database file [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-Country.mmdb]
[2022-09-19T13:11:00,406][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] successfully reloaded changed geoip database file [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-ASN.mmdb]
[2022-09-19T13:11:01,275][INFO ][o.e.i.g.DatabaseNodeService] [prod-resultselasticsearchm-101w] successfully reloaded changed geoip database file [/tmp/elasticsearch-4947918882176097480/geoip-databases/ZN56Y0twTkaQOVlD86bPDQ/GeoLite2-City.mmdb]
[root@prod-resultselasticsearchm-101w logs]#