After upgrading to 7.13.2 from 7.12.1, using systemctl to start elasticsearch fails.
[root@elastic elasticsearch]# systemctl status elasticsearch -l
ā elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: timeout) since Sun 2021-06-27 12:29:09 ADT; 29s ago
Docs: https://www.elastic.co
Process: 13959 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=143)
Main PID: 13959 (code=exited, status=143)
Jun 27 12:27:54 elastic.nbp-lab.com systemd[1]: Starting Elasticsearch...
Jun 27 12:29:09 elastic.nbp-lab.com systemd[1]: elasticsearch.service start operation timed out. Terminating.
Jun 27 12:29:09 elastic.nbp-lab.com systemd[1]: Failed to start Elasticsearch.
Jun 27 12:29:09 elastic.nbp-lab.com systemd[1]: Unit elasticsearch.service entered failed state.
Jun 27 12:29:09 elastic.nbp-lab.com systemd[1]: elasticsearch.service failed.
However, I can manually start elasticsearch by "sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch".
Below is the output from the elasticsearch logs after it fails to start:
[2021-06-27T12:28:21,763][INFO ][o.e.n.Node ] [elastic] version[7.13.2], pid[13959], build[default/rpm/4d960a0733be83dd2543ca018aa4ddc42e956800/2021-06-10T21:01:55.251515791Z],
OS[Linux/3.10.0-1160.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36]
[2021-06-27T12:28:21,779][INFO ][o.e.n.Node ] [elastic] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-06-27T12:28:21,781][INFO ][o.e.n.Node ] [elastic] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPre
Touch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.nett
y.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.p
roviders=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-9846299108287688711, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/
lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms
7942m, -Xmx7942m, -XX:MaxDirectMemorySize=4164943872, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.p
ath.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2021-06-27T12:28:33,397][INFO ][o.e.p.PluginsService ] [elastic] loaded module [aggs-matrix-stats]
[2021-06-27T12:28:33,399][INFO ][o.e.p.PluginsService ] [elastic] loaded module [analysis-common]
[2021-06-27T12:28:33,400][INFO ][o.e.p.PluginsService ] [elastic] loaded module [constant-keyword]
[2021-06-27T12:28:33,401][INFO ][o.e.p.PluginsService ] [elastic] loaded module [frozen-indices]
[2021-06-27T12:28:33,403][INFO ][o.e.p.PluginsService ] [elastic] loaded module [ingest-common]
[2021-06-27T12:28:33,404][INFO ][o.e.p.PluginsService ] [elastic] loaded module [ingest-geoip]
[2021-06-27T12:28:33,405][INFO ][o.e.p.PluginsService ] [elastic] loaded module [ingest-user-agent]
[2021-06-27T12:28:33,406][INFO ][o.e.p.PluginsService ] [elastic] loaded module [kibana]
[2021-06-27T12:28:33,407][INFO ][o.e.p.PluginsService ] [elastic] loaded module [lang-expression]
[2021-06-27T12:28:33,408][INFO ][o.e.p.PluginsService ] [elastic] loaded module [lang-mustache]
[2021-06-27T12:28:33,409][INFO ][o.e.p.PluginsService ] [elastic] loaded module [lang-painless]
[2021-06-27T12:28:33,410][INFO ][o.e.p.PluginsService ] [elastic] loaded module [mapper-extras]
[2021-06-27T12:28:33,411][INFO ][o.e.p.PluginsService ] [elastic] loaded module [mapper-version]
[2021-06-27T12:28:33,412][INFO ][o.e.p.PluginsService ] [elastic] loaded module [parent-join]
[2021-06-27T12:28:33,413][INFO ][o.e.p.PluginsService ] [elastic] loaded module [percolator]
[2021-06-27T12:28:33,414][INFO ][o.e.p.PluginsService ] [elastic] loaded module [rank-eval]
[2021-06-27T12:28:33,415][INFO ][o.e.p.PluginsService ] [elastic] loaded module [reindex]
[2021-06-27T12:28:33,416][INFO ][o.e.p.PluginsService ] [elastic] loaded module [repositories-metering-api]
[2021-06-27T12:28:33,417][INFO ][o.e.p.PluginsService ] [elastic] loaded module [repository-encrypted]
[2021-06-27T12:28:33,418][INFO ][o.e.p.PluginsService ] [elastic] loaded module [repository-url]
[2021-06-27T12:28:33,419][INFO ][o.e.p.PluginsService ] [elastic] loaded module [runtime-fields-common]
[2021-06-27T12:28:33,420][INFO ][o.e.p.PluginsService ] [elastic] loaded module [search-business-rules]
[2021-06-27T12:28:33,421][INFO ][o.e.p.PluginsService ] [elastic] loaded module [searchable-snapshots]
[2021-06-27T12:28:33,422][INFO ][o.e.p.PluginsService ] [elastic] loaded module [snapshot-repo-test-kit]
[2021-06-27T12:28:33,423][INFO ][o.e.p.PluginsService ] [elastic] loaded module [spatial]
[2021-06-27T12:28:33,423][INFO ][o.e.p.PluginsService ] [elastic] loaded module [systemd]
[2021-06-27T12:28:33,424][INFO ][o.e.p.PluginsService ] [elastic] loaded module [transform]
[2021-06-27T12:28:33,425][INFO ][o.e.p.PluginsService ] [elastic] loaded module [transport-netty4]
[2021-06-27T12:28:33,426][INFO ][o.e.p.PluginsService ] [elastic] loaded module [unsigned-long]
[2021-06-27T12:28:33,427][INFO ][o.e.p.PluginsService ] [elastic] loaded module [vectors]
[2021-06-27T12:28:33,428][INFO ][o.e.p.PluginsService ] [elastic] loaded module [wildcard]
[2021-06-27T12:28:33,429][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-aggregate-metric]
[2021-06-27T12:28:33,431][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-analytics]
[2021-06-27T12:28:33,432][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-async]
[2021-06-27T12:28:33,433][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-async-search]
[2021-06-27T12:28:33,434][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-autoscaling]
[2021-06-27T12:28:33,435][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-ccr]
[2021-06-27T12:28:33,436][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-core]
[2021-06-27T12:28:33,437][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-data-streams]
[2021-06-27T12:28:33,438][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-deprecation]
[2021-06-27T12:28:33,440][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-enrich]
[2021-06-27T12:28:33,441][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-eql]
[2021-06-27T12:28:33,442][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-fleet]
[2021-06-27T12:28:33,443][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-graph]
[2021-06-27T12:28:33,444][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-identity-provider]
[2021-06-27T12:28:33,445][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-ilm]
[2021-06-27T12:28:33,446][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-logstash]
[2021-06-27T12:28:33,447][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-ml]
[2021-06-27T12:28:33,449][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-monitoring]
[2021-06-27T12:28:33,450][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-ql]
[2021-06-27T12:28:33,451][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-rollup]
[2021-06-27T12:28:33,452][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-security]
[2021-06-27T12:28:33,453][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-shutdown]
[2021-06-27T12:28:33,454][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-sql]
[2021-06-27T12:28:33,455][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-stack]
[2021-06-27T12:28:33,456][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-text-structure]
[2021-06-27T12:28:33,458][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-voting-only-node]
[2021-06-27T12:28:33,459][INFO ][o.e.p.PluginsService ] [elastic] loaded module [x-pack-watcher]
[2021-06-27T12:28:33,461][INFO ][o.e.p.PluginsService ] [elastic] no plugins loaded
[2021-06-27T12:28:33,638][INFO ][o.e.e.NodeEnvironment ] [elastic] using [1] data paths, mounts [[/ (/dev/mapper/centos_elastic-root)]], net usable_space [487.5gb], net total_space [491gb
], types [xfs]
[2021-06-27T12:28:33,640][INFO ][o.e.e.NodeEnvironment ] [elastic] heap size [7.7gb], compressed ordinary object pointers [true]
[2021-06-27T12:28:34,167][INFO ][o.e.n.Node ] [elastic] node name [elastic], node ID [xLRTCPa6SpC8rzojZ_kMQw], cluster name [antsle-cluster], roles [transform, data_frozen, mas
ter, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-06-27T12:28:58,277][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elastic] [controller/14184] [Main.cc@117] controller (64 bit): Version 7.13.2 (Build 4d6c6f14d75f39) Copyright (c) 2021 El
asticsearch BV
[2021-06-27T12:28:59,767][INFO ][o.e.x.s.a.s.FileRolesStore] [elastic] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-06-27T12:29:02,255][INFO ][o.e.i.g.LocalDatabases ] [elastic] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] an
d watching [/etc/elasticsearch/ingest-geoip] for changes
[2021-06-27T12:29:02,260][INFO ][o.e.i.g.DatabaseRegistry ] [elastic] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-9846299108287688711/geoip-databases/x
LRTCPa6SpC8rzojZ_kMQw]
[2021-06-27T12:29:05,325][INFO ][o.e.t.NettyAllocator ] [elastic] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_alloca
tion_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-06-27T12:29:05,721][INFO ][o.e.d.DiscoveryModule ] [elastic] using discovery type [single-node] and seed hosts providers [settings]
[2021-06-27T12:29:08,101][INFO ][o.e.g.DanglingIndicesState] [elastic] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and m
ust be managed manually
[2021-06-27T12:29:09,508][INFO ][o.e.x.m.p.NativeController] [elastic] Native controller process has stopped - no new native processes can be started
Thanks.