Unable to start elasticsearch as service

Elastic Stack Elasticsearch
1

Elasticsearch is unable to start:

these are the elasticsearch logs:

            at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
            at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:212) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.node.Node.<init>(Node.java:271) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.0.jar:7.0.0]
            ... 6 more

and these are the elasticsearch service status:

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2019-04-25 04:56:30 UTC; 2s ago
     Docs: http://www.elastic.co
  Process: 18352 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 18352 (code=exited, status=1/FAILURE)

Apr 25 04:56:28 mastertesting systemd[1]: Started Elasticsearch.
Apr 25 04:56:28 mastertesting elasticsearch[18352]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Apr 25 04:56:30 mastertesting systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Apr 25 04:56:30 mastertesting systemd[1]: Unit elasticsearch.service entered failed state.
Apr 25 04:56:30 mastertesting systemd[1]: elasticsearch.service failed.

Plz reply as soon as possible I'm unable to fix the problem.

2

HI @Kajol_Nimesh,

Could you please paste full elasticsearch logs and elasticsearch.yml file here to identify the problem.

Regards,
Harsh Bajaj

3 
   Elasticsearch logs:

    [2019-04-25T05:20:57,831][ERROR][o.e.b.Bootstrap          ] [mastertesting] Exception
    java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
            at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:297) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.node.Node.<init>(Node.java:271) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.0.0.jar:7.0.0]
            at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.0.0.jar:7.0.0]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.0.0.jar:7.0.0]
    Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0
            at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:219) ~[elasticsearch-7.0.0.jar:7.0.0]
            at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-7.0.0.jar:7.0.0]
            ... 12 more
4 
 Caused by: java.nio.file.AccessDeniedException: /var/lib/elasticsearch/nodes/0/node.lock
                at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
                at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
                at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
                at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182) ~[?:?]
                at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]
                at java.nio.channels.FileChannel.open(FileChannel.java:345) ~[?:?]
                at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
                at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
                at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
                at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:212) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-7.0.0.jar:7.0.0]
                ... 12 more
        [2019-04-25T05:20:57,843][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [mastertesting] uncaught exception in thread [main]
        org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
                at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.0.0.jar:7.0.0]
                at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.0.0.jar:7.0.0]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.0.0.jar:7.0.0]
        Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
                at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:297) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.node.Node.<init>(Node.java:271) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.0.jar:7.0.0]
                ... 6 more
        Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0
                at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:219) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:267) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.node.Node.<init>(Node.java:271) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.0.0.jar:7.0.0]
                at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.0.0.jar:7.0.0]
5

Elasticsearch.yml file

#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: localhost
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
"/etc/elasticsearch/elasticsearch.yml" 88L, 2843C
6

Hi @Kajol_Nimesh,

Access denied exception is showing for the path /var/lib/elasticsearch. could you please check the access permission for this path.

If you see below error it clearly show that not writable and multiple nodes were started. Please check and let me know if you still facing any issue.

Regards,
Harsh Bajaj

7

Error solved it was permission issue. I changed it by giving var/lib/elasticsearch the permission to root by:

sudo chown -R elasticsearch:elasticsearch /var/lib/elasticsearch

Thank You,
Regards,
Kajol Nimesh

4 Likes
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.