Ubuntu 20.04 tar/zip installed - systemd timeout

Hello,

I have a strange issue with a fresh tar/zip installation of Elasticsearch. When I use Elasticsearch v6 this way - there is no problem. Also apt installation works fine. But when I want to use systemd with Elasticsearch v7.x I got systemd timeouts.

systemctl restart elasticsearch7.service output:

Job for elasticsearch7.service failed because a timeout was exceeded.
See "systemctl status elasticsearch7.service" and "journalctl -xe" for details.

journalctl -xe output:

Oct 18 15:26:04 h1 systemd[1]: elasticsearch7.service: start operation timed out. Terminating.
Oct 18 15:26:05 h1 systemd[1]: elasticsearch7.service: Failed with result 'timeout'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit elasticsearch7.service has entered the 'failed' state with result 'timeout'.
Oct 18 15:26:05 h1 systemd[1]: Failed to start Elasticsearch.
-- Subject: A start job for unit elasticsearch7.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit elasticsearch7.service has finished with a failure.
-- 
-- The job identifier is 47534 and the job result is failed.

config/Elasticsearch.yml:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: h1cluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: h1
node.master: true
node.data: true
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 127.0.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9500
transport.port: 9501
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["127.0.0.1"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["h1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

/lib/systemd/system/elasticsearch7.service:

[Unit]
Description=Elasticsearch
Documentation=https://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
RuntimeDirectory=elasticsearch
PrivateTmp=true
Environment=ES_HOME=/home/imper/elasticsearch7
Environment=ES_PATH_CONF=/home/imper/elasticsearch7/config
Environment=PID_DIR=/var/run/elasticsearch
Environment=ES_SD_NOTIFY=true
EnvironmentFile=-/etc/default/elasticsearch7

WorkingDirectory=/home/imper/elasticsearch7

User=imper
Group=imper

ExecStart=/home/imper/elasticsearch7/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch7.pid --quiet

# StandardOutput is configured to redirect to journalctl since
# some error messages may be logged in standard output before
# elasticsearch logging system is initialized. Elasticsearch
# stores its logs in /var/log/elasticsearch and does not use
# journalctl by default. If you also want to enable journalctl
# logging, you can simply remove the "quiet" option from ExecStart.
StandardOutput=journal
StandardError=inherit

#dodalem sam
LimitMEMLOCK=infinity

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65535

# Specifies the maximum number of processes
LimitNPROC=4096

# Specifies the maximum size of virtual memory
LimitAS=infinity

# Specifies the maximum file size
LimitFSIZE=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM

# Send the signal only to the JVM rather than its control group
KillMode=process

# Java process is never killed
SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

# Allow a slow startup before the systemd notifier module kicks in to extend the timeout
TimeoutStartSec=180

[Install]
WantedBy=multi-user.target

# Built for packages-7.15.1 (packages)

/etc/default/elasticsearch7:

################################
# Elasticsearch
################################

# Elasticsearch home directory
#ES_HOME=/usr/share/elasticsearch

# Elasticsearch Java path
#ES_JAVA_HOME=

# Elasticsearch configuration directory
# Note: this setting will be shared with command-line tools
ES_PATH_CONF=/home/imper/elasticsearch7/config

# Elasticsearch PID directory
#PID_DIR=/var/run/elasticsearch

# Additional Java OPTS
ES_JAVA_OPTS="-Xms4g -Xmx4g"

# Configure restart on package upgrade (true, every other setting will lead to not restarting)
#RESTART_ON_UPGRADE=true

################################
# Elasticsearch service
################################

# SysV init.d
#
# The number of seconds to wait before checking if Elasticsearch started successfully as a daemon pr>
ES_STARTUP_SLEEP_TIME=5

################################
# System properties
################################

# Specifies the maximum file descriptor number that can be opened by this process
# When using Systemd, this setting is ignored and the LimitNOFILE defined in
# /usr/lib/systemd/system/elasticsearch.service takes precedence
#MAX_OPEN_FILES=65535

# The maximum number of bytes of memory that may be locked into RAM
# Set to "unlimited" if you use the 'bootstrap.memory_lock: true' option
# in elasticsearch.yml.
# When using systemd, LimitMEMLOCK must be set in a unit file such as
# /etc/systemd/system/elasticsearch.service.d/override.conf.
MAX_LOCKED_MEMORY=unlimited

# Maximum number of VMA (Virtual Memory Areas) a process can own
# When using Systemd, this setting is ignored and the 'vm.max_map_count'
# property is set at boot time in /usr/lib/sysctl.d/elasticsearch.conf
#MAX_MAP_COUNT=262144

I really don't get it. When i run ./bin/Elasticsearch it works, but systemd seems to think it didn't start. Also when systemd is before a timeout I get a normal response from GET /.

Any ideas? Tried TimeoutStartUSec=15min and it didn't worked for me :slight_smile:

logs/h1cluster.log:

[2021-10-18T15:23:06,777][INFO ][o.e.n.Node               ] [h1] version[7.15.1], pid[2905782], build[default/tar/83c34f456ae29d60e94d886e455e6a3409bba9ed/2021-10-07T21:56:19.031608185Z], OS[Linux/5.4.0-88-generic/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17/17+35]
[2021-10-18T15:23:06,780][INFO ][o.e.n.Node               ] [h1] JVM home [/home/imper/elasticsearch7/jdk], using bundled JDK [true]
[2021-10-18T15:23:06,781][INFO ][o.e.n.Node               ] [h1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-5250825756931447202, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms4g, -Xmx4g, -XX:MaxDirectMemorySize=2147483648, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/home/imper/elasticsearch7, -Des.path.conf=/home/imper/elasticsearch7/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]

...plugins loaded here...

[2021-10-18T15:23:08,115][INFO ][o.e.p.PluginsService     ] [h1] no plugins loaded
[2021-10-18T15:23:08,132][INFO ][o.e.e.NodeEnvironment    ] [h1] using [1] data paths, mounts [[/ (/dev/md2)]], net usable_space [2.9tb], net total_space [3.4tb], types [ext4]
[2021-10-18T15:23:08,133][INFO ][o.e.e.NodeEnvironment    ] [h1] heap size [4gb], compressed ordinary object pointers [true]
[2021-10-18T15:23:08,170][INFO ][o.e.n.Node               ] [h1] node name [h1], node ID [3t3_P9cUSl6lGp5yp7EnIA], cluster name [h1cluster], roles [transform, data_frozen, master, remote_cluster_client, data, data_content, data_hot, data_warm, data_cold, ingest]
[2021-10-18T15:23:10,571][INFO ][o.e.x.s.a.s.FileRolesStore] [h1] parsed [0] roles from file [/home/imper/elasticsearch7/config/roles.yml]
[2021-10-18T15:23:10,920][INFO ][o.e.i.g.LocalDatabases   ] [h1] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/home/imper/elasticsearch7/config/ingest-geoip] for changes
[2021-10-18T15:23:10,920][INFO ][o.e.i.g.DatabaseRegistry ] [h1] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-5250825756931447202/geoip-databases/3t3_P9cUSl6lGp5yp7EnIA]
[2021-10-18T15:23:11,211][INFO ][o.e.t.NettyAllocator     ] [h1] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-10-18T15:23:11,247][INFO ][o.e.d.DiscoveryModule    ] [h1] using discovery type [zen] and seed hosts providers [settings]
[2021-10-18T15:23:11,475][INFO ][o.e.g.DanglingIndicesState] [h1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-10-18T15:23:11,673][INFO ][o.e.n.Node               ] [h1] initialized
[2021-10-18T15:23:11,673][INFO ][o.e.n.Node               ] [h1] starting ...
[2021-10-18T15:23:11,680][INFO ][o.e.x.s.c.f.PersistentCache] [h1] persistent cache index loaded
[2021-10-18T15:23:11,733][INFO ][o.e.t.TransportService   ] [h1] publish_address {127.0.0.1:9501}, bound_addresses {127.0.0.1:9501}
[2021-10-18T15:23:11,879][INFO ][o.e.c.c.Coordinator      ] [h1] cluster UUID [eEgmxBULQ5Kab7EBA3JHHQ]
[2021-10-18T15:23:12,005][INFO ][o.e.c.s.MasterService    ] [h1] elected-as-master ([1] nodes joined)[{h1}{3t3_P9cUSl6lGp5yp7EnIA}{uUPYjIlXQOCaYvtUMQwNsQ}{127.0.0.1}{127.0.0.1:9501}{cdfhimrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 6, version: 83, delta: master node changed {previous [], current [{h1}{3t3_P9cUSl6lGp5yp7EnIA}{uUPYjIlXQOCaYvtUMQwNsQ}{127.0.0.1}{127.0.0.1:9501}{cdfhimrstw}]}
[2021-10-18T15:23:12,042][INFO ][o.e.c.s.ClusterApplierService] [h1] master node changed {previous [], current [{h1}{3t3_P9cUSl6lGp5yp7EnIA}{uUPYjIlXQOCaYvtUMQwNsQ}{127.0.0.1}{127.0.0.1:9501}{cdfhimrstw}]}, term: 6, version: 83, reason: Publication{term=6, version=83}
[2021-10-18T15:23:12,066][INFO ][o.e.h.AbstractHttpServerTransport] [h1] publish_address {127.0.0.1:9500}, bound_addresses {127.0.0.1:9500}
[2021-10-18T15:23:12,067][INFO ][o.e.n.Node               ] [h1] started
[2021-10-18T15:23:12,192][INFO ][o.e.l.LicenseService     ] [h1] license [dd294587-85fc-47a3-9cd0-eed15cb0919c] mode [basic] - valid
[2021-10-18T15:23:12,193][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [h1] Active license is now [BASIC]; Security is disabled
[2021-10-18T15:23:12,193][WARN ][o.e.x.s.s.SecurityStatusChangeListener] [h1] Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.15/security-minimal-setup.html to enable security.
[2021-10-18T15:23:12,195][INFO ][o.e.g.GatewayService     ] [h1] recovered [1] indices into cluster_state
[2021-10-18T15:23:12,339][INFO ][o.e.i.g.GeoIpDownloader  ] [h1] updating geoip databases
[2021-10-18T15:23:12,339][INFO ][o.e.i.g.GeoIpDownloader  ] [h1] fetching geoip databases overview from [https://geoip.elastic.co/v1/database?elastic_geoip_service_tos=agree]
[2021-10-18T15:23:12,358][INFO ][o.e.c.r.a.AllocationService] [h1] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.geoip_databases][0]]]).
[2021-10-18T15:23:12,373][INFO ][o.e.i.g.DatabaseRegistry ] [h1] downloading geoip database [GeoLite2-Country.mmdb] to [/tmp/elasticsearch-5250825756931447202/geoip-databases/3t3_P9cUSl6lGp5yp7EnIA/GeoLite2-Country.mmdb.tmp.gz]
[2021-10-18T15:23:12,374][INFO ][o.e.i.g.DatabaseRegistry ] [h1] downloading geoip database [GeoLite2-ASN.mmdb] to [/tmp/elasticsearch-5250825756931447202/geoip-databases/3t3_P9cUSl6lGp5yp7EnIA/GeoLite2-ASN.mmdb.tmp.gz]
[2021-10-18T15:23:12,374][INFO ][o.e.i.g.DatabaseRegistry ] [h1] downloading geoip database [GeoLite2-City.mmdb] to [/tmp/elasticsearch-5250825756931447202/geoip-databases/3t3_P9cUSl6lGp5yp7EnIA/GeoLite2-City.mmdb.tmp.gz]
[2021-10-18T15:23:12,517][INFO ][o.e.i.g.DatabaseRegistry ] [h1] successfully reloaded changed geoip database file [/tmp/elasticsearch-5250825756931447202/geoip-databases/3t3_P9cUSl6lGp5yp7EnIA/GeoLite2-Country.mmdb]
[2021-10-18T15:23:12,533][INFO ][o.e.i.g.DatabaseRegistry ] [h1] successfully reloaded changed geoip database file [/tmp/elasticsearch-5250825756931447202/geoip-databases/3t3_P9cUSl6lGp5yp7EnIA/GeoLite2-ASN.mmdb]
[2021-10-18T15:23:12,921][INFO ][o.e.i.g.GeoIpDownloader  ] [h1] geoip database [GeoLite2-ASN.mmdb] is up to date, updated timestamp
[2021-10-18T15:23:12,945][INFO ][o.e.i.g.GeoIpDownloader  ] [h1] geoip database [GeoLite2-City.mmdb] is up to date, updated timestamp
[2021-10-18T15:23:12,956][INFO ][o.e.i.g.DatabaseRegistry ] [h1] successfully reloaded changed geoip database file [/tmp/elasticsearch-5250825756931447202/geoip-databases/3t3_P9cUSl6lGp5yp7EnIA/GeoLite2-City.mmdb]
[2021-10-18T15:23:12,963][INFO ][o.e.i.g.GeoIpDownloader  ] [h1] geoip database [GeoLite2-Country.mmdb] is up to date, updated timestamp
[2021-10-18T15:26:04,979][INFO ][o.e.n.Node               ] [h1] stopping ...
[2021-10-18T15:26:04,981][INFO ][o.e.x.w.WatcherService   ] [h1] stopping watch service, reason [shutdown initiated]
[2021-10-18T15:26:04,981][INFO ][o.e.x.w.WatcherLifeCycleService] [h1] watcher has stopped and shutdown
[2021-10-18T15:26:05,461][INFO ][o.e.n.Node               ] [h1] stopped
[2021-10-18T15:26:05,461][INFO ][o.e.n.Node               ] [h1] closing ...
[2021-10-18T15:26:05,469][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [h1] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-5250825756931447202/geoip-databases/3t3_P9cUSl6lGp5yp7EnIA/GeoLite2-Country.mmdb]
[2021-10-18T15:26:05,469][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [h1] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-5250825756931447202/geoip-databases/3t3_P9cUSl6lGp5yp7EnIA/GeoLite2-ASN.mmdb]
[2021-10-18T15:26:05,469][INFO ][o.e.i.g.DatabaseReaderLazyLoader] [h1] evicted [0] entries from cache after reloading database [/tmp/elasticsearch-5250825756931447202/geoip-databases/3t3_P9cUSl6lGp5yp7EnIA/GeoLite2-City.mmdb]
[2021-10-18T15:26:05,470][INFO ][o.e.n.Node               ] [h1] closed

From the docs:

The Elasticsearch .tar.gz package does not include the systemd module. To manage Elasticsearch as a service, use the Debian or RPM package instead.

I mean you technically can run it with systemd as a regular process but you can't use Type=notify without the systemd module.

2 Likes

Thank you for the help. Removed Type=notify line, and now it's working :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.