Systemctl failing to start elasticsearch

bnason · June 28, 2021, 11:55am

After upgrading to 7.13.2 from 7.12.1, using systemctl to start elasticsearch fails.

[root@elastic elasticsearch]# systemctl status elasticsearch -l
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Sun 2021-06-27 12:29:09 ADT; 29s ago
     Docs: https://www.elastic.co
  Process: 13959 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=143)
 Main PID: 13959 (code=exited, status=143)

Jun 27 12:27:54 elastic.nbp-lab.com systemd[1]: Starting Elasticsearch...
Jun 27 12:29:09 elastic.nbp-lab.com systemd[1]: elasticsearch.service start operation timed out. Terminating.
Jun 27 12:29:09 elastic.nbp-lab.com systemd[1]: Failed to start Elasticsearch.
Jun 27 12:29:09 elastic.nbp-lab.com systemd[1]: Unit elasticsearch.service entered failed state.
Jun 27 12:29:09 elastic.nbp-lab.com systemd[1]: elasticsearch.service failed.

However, I can manually start elasticsearch by "sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch".

Below is the output from the elasticsearch logs after it fails to start:

[2021-06-27T12:28:21,763][INFO ][o.e.n.Node               ] [elastic] version[7.13.2], pid[13959], build[default/rpm/4d960a0733be83dd2543ca018aa4ddc42e956800/2021-06-10T21:01:55.251515791Z],
 OS[Linux/3.10.0-1160.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36]
[2021-06-27T12:28:21,779][INFO ][o.e.n.Node               ] [elastic] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-06-27T12:28:21,781][INFO ][o.e.n.Node               ] [elastic] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPre
Touch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.nett
y.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.p
roviders=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-9846299108287688711, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/
lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms
7942m, -Xmx7942m, -XX:MaxDirectMemorySize=4164943872, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.p
ath.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2021-06-27T12:28:33,397][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [aggs-matrix-stats]
[2021-06-27T12:28:33,399][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [analysis-common]
[2021-06-27T12:28:33,400][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [constant-keyword]
[2021-06-27T12:28:33,401][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [frozen-indices]
[2021-06-27T12:28:33,403][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [ingest-common]
[2021-06-27T12:28:33,404][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [ingest-geoip]
[2021-06-27T12:28:33,405][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [ingest-user-agent]
[2021-06-27T12:28:33,406][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [kibana]
[2021-06-27T12:28:33,407][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [lang-expression]
[2021-06-27T12:28:33,408][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [lang-mustache]
[2021-06-27T12:28:33,409][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [lang-painless]
[2021-06-27T12:28:33,410][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [mapper-extras]
[2021-06-27T12:28:33,411][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [mapper-version]
[2021-06-27T12:28:33,412][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [parent-join]
[2021-06-27T12:28:33,413][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [percolator]
[2021-06-27T12:28:33,414][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [rank-eval]
[2021-06-27T12:28:33,415][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [reindex]
[2021-06-27T12:28:33,416][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [repositories-metering-api]
[2021-06-27T12:28:33,417][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [repository-encrypted]
[2021-06-27T12:28:33,418][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [repository-url]
[2021-06-27T12:28:33,419][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [runtime-fields-common]
[2021-06-27T12:28:33,420][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [search-business-rules]
[2021-06-27T12:28:33,421][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [searchable-snapshots]
[2021-06-27T12:28:33,422][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [snapshot-repo-test-kit]
[2021-06-27T12:28:33,423][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [spatial]
[2021-06-27T12:28:33,423][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [systemd]
[2021-06-27T12:28:33,424][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [transform]
[2021-06-27T12:28:33,425][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [transport-netty4]
[2021-06-27T12:28:33,426][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [unsigned-long]
[2021-06-27T12:28:33,427][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [vectors]
[2021-06-27T12:28:33,428][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [wildcard]
[2021-06-27T12:28:33,429][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-aggregate-metric]
[2021-06-27T12:28:33,431][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-analytics]
[2021-06-27T12:28:33,432][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-async]
[2021-06-27T12:28:33,433][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-async-search]
[2021-06-27T12:28:33,434][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-autoscaling]
[2021-06-27T12:28:33,435][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-ccr]
[2021-06-27T12:28:33,436][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-core]
[2021-06-27T12:28:33,437][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-data-streams]
[2021-06-27T12:28:33,438][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-deprecation]
[2021-06-27T12:28:33,440][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-enrich]
[2021-06-27T12:28:33,441][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-eql]
[2021-06-27T12:28:33,442][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-fleet]
[2021-06-27T12:28:33,443][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-graph]
[2021-06-27T12:28:33,444][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-identity-provider]
[2021-06-27T12:28:33,445][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-ilm]
[2021-06-27T12:28:33,446][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-logstash]
[2021-06-27T12:28:33,447][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-ml]
[2021-06-27T12:28:33,449][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-monitoring]
[2021-06-27T12:28:33,450][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-ql]
[2021-06-27T12:28:33,451][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-rollup]
[2021-06-27T12:28:33,452][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-security]
[2021-06-27T12:28:33,453][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-shutdown]
[2021-06-27T12:28:33,454][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-sql]
[2021-06-27T12:28:33,455][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-stack]
[2021-06-27T12:28:33,456][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-text-structure]
[2021-06-27T12:28:33,458][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-voting-only-node]
[2021-06-27T12:28:33,459][INFO ][o.e.p.PluginsService     ] [elastic] loaded module [x-pack-watcher]
[2021-06-27T12:28:33,461][INFO ][o.e.p.PluginsService     ] [elastic] no plugins loaded
[2021-06-27T12:28:33,638][INFO ][o.e.e.NodeEnvironment    ] [elastic] using [1] data paths, mounts [[/ (/dev/mapper/centos_elastic-root)]], net usable_space [487.5gb], net total_space [491gb
], types [xfs]
[2021-06-27T12:28:33,640][INFO ][o.e.e.NodeEnvironment    ] [elastic] heap size [7.7gb], compressed ordinary object pointers [true]
[2021-06-27T12:28:34,167][INFO ][o.e.n.Node               ] [elastic] node name [elastic], node ID [xLRTCPa6SpC8rzojZ_kMQw], cluster name [antsle-cluster], roles [transform, data_frozen, mas
ter, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-06-27T12:28:58,277][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elastic] [controller/14184] [Main.cc@117] controller (64 bit): Version 7.13.2 (Build 4d6c6f14d75f39) Copyright (c) 2021 El
asticsearch BV
[2021-06-27T12:28:59,767][INFO ][o.e.x.s.a.s.FileRolesStore] [elastic] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-06-27T12:29:02,255][INFO ][o.e.i.g.LocalDatabases   ] [elastic] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] an
d watching [/etc/elasticsearch/ingest-geoip] for changes
[2021-06-27T12:29:02,260][INFO ][o.e.i.g.DatabaseRegistry ] [elastic] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-9846299108287688711/geoip-databases/x
LRTCPa6SpC8rzojZ_kMQw]
[2021-06-27T12:29:05,325][INFO ][o.e.t.NettyAllocator     ] [elastic] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_alloca
tion_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-06-27T12:29:05,721][INFO ][o.e.d.DiscoveryModule    ] [elastic] using discovery type [single-node] and seed hosts providers [settings]
[2021-06-27T12:29:08,101][INFO ][o.e.g.DanglingIndicesState] [elastic] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and m
ust be managed manually
[2021-06-27T12:29:09,508][INFO ][o.e.x.m.p.NativeController] [elastic] Native controller process has stopped - no new native processes can be started

Thanks.

warkolm · June 29, 2021, 1:51am

Is there anything in your OS logs at the time you are starting Elasticsearch?

bnason · June 29, 2021, 11:14am

The only thing I see in the /var/log/messages is:

Jun 27 12:27:54 elastic systemd: Starting Elasticsearch...
Jun 27 12:29:09 elastic systemd: elasticsearch.service start operation timed out. Terminating.
Jun 27 12:29:09 elastic systemd: Failed to start Elasticsearch.
Jun 27 12:29:09 elastic systemd: Unit elasticsearch.service entered failed state.
Jun 27 12:29:09 elastic systemd: elasticsearch.service failed.

warkolm · July 1, 2021, 1:55am

That usually indicates something closed Elasticsearch. And the output from systemd seems to indicate it timed out waiting for Elasticsearch to start. Can you increase the timeout that systemd waits for for this service?

bnason · July 5, 2021, 12:18pm

Thanks, that was it. I changed TimeoutStartSec=75 to 180 and the service start ok.

system · August 2, 2021, 12:18pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.