Elasticsearch can not be started by service

I installed an elasticsearch 7.10.0 by tar.gz archives because my network is a closed network. I can run it as a deamon with ' ../bin/elasticsearch -d -p es.pid' but systemctl command can not be run.
The followings are my server setting, service file and logs
-------------------------------------------------------------------------------------
* Server

  • CentOS 7.4
  • Cloud Server

*Result of 'systemctl status elasticsearc.service'
● elasticsearch.service - ElasticSearch 7.10.0
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: timeout) since Thu 2020-12-03 15:05:52 KST; 3s ago
Docs: https://www.elastic.co
Process: 25446 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=143)
Main PID: 25446 (code=exited, status=143)

Dec 03 15:04:37 p-frida-pk1-e03 systemd[1]: Starting ElasticSearch 7.10.0...
Dec 03 15:05:52 p-frida-pk1-e03 systemd[1]: elasticsearch.service start operation timed out. Terminating.
Dec 03 15:05:52 p-frida-pk1-e03 systemd[1]: Failed to start ElasticSearch 7.10.0.
Dec 03 15:05:52 p-frida-pk1-e03 systemd[1]: Unit elasticsearch.service entered failed state.
Dec 03 15:05:52 p-frida-pk1-e03 systemd[1]: elasticsearch.service failed.

*Error Log
...
[2020-12-02T17:51:58,699][INFO ][o.e.p.PluginsService ] [node-1] no plugins loaded
[2020-12-02T17:51:58,774][INFO ][o.e.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/data (10.220.190.107:/p_frida_pk1_nas)]], net usable_space [4.3tb], net total_space [4.3tb], types [nfs]
[2020-12-02T17:51:58,775][INFO ][o.e.e.NodeEnvironment ] [node-1] heap size [1gb], compressed ordinary object pointers [true]
[2020-12-02T17:51:59,097][INFO ][o.e.n.Node ] [node-1] node name [node-1], node ID [aRZHNwmsS32V7tvXL1ysFQ], cluster name [elk-server], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2020-12-02T17:52:03,325][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/1707] [Main.cc@114] controller (64 bit): Version 7.10.0 (Build ac991e2e31f99d) Copyright (c) 2020 Elasticsearch BV
[2020-12-02T17:52:03,972][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2020-12-02T17:52:05,560][INFO ][o.e.t.NettyAllocator ] [node-1] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2020-12-02T17:52:05,660][INFO ][o.e.d.DiscoveryModule ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2020-12-02T17:52:06,084][WARN ][o.e.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2020-12-02T17:52:06,474][INFO ][o.e.n.Node ] [node-1] initialized
[2020-12-02T17:52:06,474][INFO ][o.e.n.Node ] [node-1] starting ...
[2020-12-02T17:52:06,724][INFO ][o.e.t.TransportService ] [node-1] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2020-12-02T17:52:07,207][WARN ][o.e.b.BootstrapChecks ] [node-1] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2020-12-02T17:52:07,207][WARN ][o.e.b.BootstrapChecks ] [node-1] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-12-02T17:52:07,209][INFO ][o.e.c.c.Coordinator ] [node-1] cluster UUID [GgY95KdmRQi7Y0VHz3Hbmw]
[2020-12-02T17:52:07,227][INFO ][o.e.c.c.ClusterBootstrapService] [node-1] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2020-12-02T17:52:07,368][INFO ][o.e.c.s.MasterService ] [node-1] elected-as-master ([1] nodes joined)[{node-1}{aRZHNwmsS32V7tvXL1ysFQ}{KoQOdrzoT_CCEwFUpzTsdg}{127.0.0.1}{127.0.0.1:9300}{cdhilmrstw}{ml.machine_memory=25111715840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, BECOME_MASTER_TASK, FINISH_ELECTION], term: 13, version: 182, delta: master node changed {previous , current [{node-1}{aRZHNwmsS32V7tvXL1ysFQ}{KoQOdrzoT_CCEwFUpzTsdg}{127.0.0.1}{127.0.0.1:9300}{cdhilmrstw}{ml.machine_memory=25111715840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}
[2020-12-02T17:52:07,517][INFO ][o.e.c.s.ClusterApplierService] [node-1] master node changed {previous , current [{node-1}{aRZHNwmsS32V7tvXL1ysFQ}{KoQOdrzoT_CCEwFUpzTsdg}{127.0.0.1}{127.0.0.1:9300}{cdhilmrstw}{ml.machine_memory=25111715840, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 13, version: 182, reason: Publication{term=13, version=182}
[2020-12-02T17:52:07,620][INFO ][o.e.h.AbstractHttpServerTransport] [node-1] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-12-02T17:52:07,621][INFO ][o.e.n.Node ] [node-1] started
[2020-12-02T17:52:07,884][INFO ][o.e.l.LicenseService ] [node-1] license [f93e3bd4-e269-4db1-a5fc-65ddaa48e7d0] mode [basic] - valid
[2020-12-02T17:52:07,886][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Active license is now [BASIC]; Security is disabled
[2020-12-02T17:52:07,892][INFO ][o.e.g.GatewayService ] [node-1] recovered [6] indices into cluster_state
[2020-12-02T17:52:09,684][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[ilm-history-3-000001][0]]]).
[2020-12-02T17:53:00,392][INFO ][o.e.n.Node ] [node-1] stopping ...
[2020-12-02T17:53:00,396][INFO ][o.e.x.w.WatcherService ] [node-1] stopping watch service, reason [shutdown initiated]
[2020-12-02T17:53:00,396][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/1707] [Main.cc@154] ML controller exiting
[2020-12-02T17:53:00,397][INFO ][o.e.x.w.WatcherLifeCycleService] [node-1] watcher has stopped and shutdown
[2020-12-02T17:53:00,397][INFO ][o.e.x.m.p.NativeController] [node-1] Native controller process has stopped - no new native processes can be started
[2020-12-02T17:53:00,639][INFO ][o.e.n.Node ] [node-1] stopped
[2020-12-02T17:53:00,639][INFO ][o.e.n.Node ] [node-1] closing ...
[2020-12-02T17:53:00,651][INFO ][o.e.n.Node ] [node-1] closed

  • Elasticsearch service File
    [Unit]
    Description=ElasticSearch 7.10.0
    Documentation=https://www.elastic.co
    Wants=network-online.target
    After=network-online.target

[Service]
Type=notify
RuntimeDirectory=elasticsearch
PrivateTmp=true
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/etc/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
Environment=ES_SD_NOTIFY=true
EnvironmentFile=-/etc/sysconfig/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=opr
Group=opr

ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet

# StandardOutput is configured to redirect to journalctl since
# some error messages may be logged in standard output before
# elasticsearch logging system is initialized. Elasticsearch
# stores its logs in /var/log/elasticsearch and does not use
# journalctl by default. If you also want to enable journalctl
# logging, you can simply remove the "quiet" option from ExecStart.

StandardOutput=journal
StandardError=inherit

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65535

# Specifies the maximum number of processes
LimitNPROC=4096

# Specifies the maximum size of virtual memory
LimitAS=infinity

# Specifies the maximum file size
LimitFSIZE=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM

# Send the signal only to the JVM rather than its control group
KillMode=process

# Java process is never killed
SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

# Allow a slow startup before the systemd notifier module kicks in to extend the timeout
TimeoutStartSec=75

[Install]
WantedBy=multi-user.target

# Built for {project.name}-{project.version} (${project.name})
------------------------------------------------------------------------------------------------
Owner of related directories is 'opr'. Can you help me?

When you start from the command line, are your root? If so, you probably get unlimited resources. Systemd will usually use a different id that may be subject to limits.

Some doc is in this part https://www.elastic.co/guide/en/elasticsearch/reference/current/file-descriptors.html#file-descriptors
But I don't see memory limits right now, but I'm betting it's a ulimit thing.

Thanks for your reply.
Elasticsearch from the command line is started by id 'opr', not 'root'.
I added the following line to this file'/etc/security/limits.conf'

elasticsearch - nofile 65535

and added this file '/etc/systemd/system/elasticsearch.service.d/override.conf' with the following lines.

[Service]
LimitMEMLOCK=infinity

But result is same.

Hi,
When you install elasticsearch with a plain tarball, you have to configure manually the virtual memory setting as per: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/vm-max-map-count.html
Besides, you have the bootstrap checks running, their role is to help elastic from running if not properly configured (for safety reasons). If your goal is to run a single-node install, check your elasticsearch.yml and set discovery-type to single-node as per https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.