Unable to start elasticsearch in Rhel 6

bini · January 16, 2017, 1:13pm

I am unable to start elasticsearch 5.1.1 on Rhel 6. I am getting the following error :

2017-01-16 12:20:10,344 main ERROR RollingFileManager (/var/log/elasticsearch/clarity.log) java.io.FileNotFoundException: /var/log/elasticsearch/clarity.log (Permission denied) java.io.FileNotFoundException: /var/log/elasticsearch/clarity.log (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at java.io.FileOutputStream.(FileOutputStream.java:133)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:474)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager$RollingFileManagerFactory.createManager(RollingFileManager.java:445)
at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:112)
at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:114)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.getFileManager(RollingFileManager.java:128)
at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:135)
at org.apache.logging.log4j.core.appender.RollingFileAppender$Builder.build(RollingFileAppender.java:58)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:882)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:874)

Kindly help me out

jasontedor · January 16, 2017, 1:26pm

I think that you installed Elasticsearch via the RPM distribution, but are attempting to start Elasticsearch using the script /usr/share/elasticsearch/bin/elasticsearch. However, the RPM is designed to be used as a service so you should use the service manager (I think RHEL 6 is Sys V init) on your system to start and stop Elasticsearch. If you do not want to use a service manager, you should download an archive distribution of Elasticsearch (the tar.gz or zip distribution).

bini · January 17, 2017, 8:18am

@jasontedor yes i have installed Elasticsearch via RPM distribution and i am starting the elasticsearch as a service, but i am getting the same error.

jasontedor · January 17, 2017, 10:50am

So let's look at what is happening here.

You are trying to start Elasticsearch.

The stacktrace indicates we are trying to open a log file.

You haven't indicated what the name of your cluster is. Is it clarity? If so, I suppose this is during startup. It would be easier to tell if you provided the entire stacktrace. If it's not clarity, what does clarity mean to you?

Either way, the error message that you are seeing indicates that you do not have permissions to access the log file.

When the service is installed, /var/log/elasticsearch is created with user elasticsearch as the owner and all log files created under there will have the same ownership.

If you start as the service, Elasticsearch is running as the elasticsearch user. That is why I say that it appears you are not running as the service (because then you would be running as a user other than the elasticsearch user and that user probably does not have permissions to access this directory). That is what these situations amount to the vast majority of the time.

You're telling me that this not the case here, yet you are still seeing the permissions problem.

If what you say is correct (that you are indeed running as the installed service), please check the permissions of the entire tree starting at /var/log/elasticsearch.

Rinat · January 17, 2017, 5:16pm

I clearly understand what you are saying. I, however, clearly do not understand where I'm going wrong.

sudo yum install elasticsearch - this is the command I ran to install

sudo -i service elasticsearch start - this is the command I ran to start

drw-rw-rw-. 2 elasticsearch elasticsearch 4096 Jan 11 15:21 elasticsearch -- these are the permissions to the directory

Starting elasticsearch: 2017-01-17 12:12:41,671 main ERROR Unable to create file /var/log/elasticsearch/elasticsearch.log java.io.IOException: Permission denied -- this is the first line of the error log.

Moreso, this is the permissions to the /var/log directory

drw-rw-rw-. 18 root root 4096 Jan 17 11:18 log

And the worst part of all of it is that it was working. I then have rebooted the machine when it came back up logstash continued to respawning non-stop and elasticsearch would not start.

And I'm clueless here.

Thanks for help in advance.

jasontedor · January 17, 2017, 6:18pm

Why does no one have execute permissions on the /var/log (or /var/log/elasticsearch for that matter)? If you do not have execute permissions on a directory, you can not access any files inside the directory.

Rinat · January 17, 2017, 7:54pm

All I can do is a "face palm" and "go take Linux 101". I apparently overlooked that part on /var/log.

Thanks for prompt and direct response!

jasontedor · January 17, 2017, 8:11pm

You're welcome.

bini · January 18, 2017, 6:30am

@jasontedor yes the cluster is clarity, i have changed the permissions and now i am not getting the permissions issue but now when i am trying to start elasticsearch i am getting the following error :

"elasticsearch dead but subsys locked "

i checked the logs and i got this :

[2017-01-18T06:56:18,222][ERROR][o.e.b.Bootstrap ] [clar-1] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]

I had changed the limit to 2048, but still i am getting the same error.

i have also checked the memory by doing df -h

[root@gdclgdvaplv01d ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-rootlv
9.8G 3.4G 5.9G 37% /
tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 477M 94M 358M 21% /boot
/dev/mapper/rootvg-homelv
7.8G 5.6G 1.9G 75% /home
/dev/mapper/rootvg-optlv
4.8G 1.7G 3.0G 36% /opt
/dev/mapper/rootvg-tmplv
4.8G 81M 4.5G 2% /tmp
/dev/mapper/rootvg-usrlv
5.5G 3.9G 1.4G 75% /usr
/dev/mapper/rootvg-varlv
8.5G 1.6G 6.6G 19% /var

so there is no space issue as well, kindly help me out. I am clueless about what to do to make elasticsearch running again.

jasontedor · January 18, 2017, 8:15am

If you're still getting the error message then you did not change the limit correctly. What steps did you take?

bini · January 18, 2017, 8:22am

I have done this :
[root@gdclgdvaplv01d ~]# ulimit -u 2048
[root@gdclgdvaplv01d ~]# tail /etc/security/limits.conf
nproc 2048
[root@gdclgdvaplv01d ~]#

[root@gdclgdvaplv01d ~]# chkconfig --list | grep -i Elasticsearch
elasticsearch 0:off 1:off 2:on 3:on 4:on 5:on 6:off
You have mail in /var/spool/mail/root
[root@gdclgdvaplv01d ~]# chkconfig --list | grep -i kibana
[root@gdclgdvaplv01d ~]#

jasontedor · January 18, 2017, 8:47am

That's not right. It should be something like

* - nproc 2048

bini · January 19, 2017, 10:43am

i did the nproc settings as u said

#* soft core 0
#* hard rss 10000
#@student hard nproc 20
#@faculty soft nproc 20
#@faculty hard nproc 50
#ftp hard nproc 0
#@student - maxlogins 4

- nproc 2048

then i started the elasticsearch , i got the same error again:

[claritygbl@gdclgdvaplv01d elasticsearch]$ sudo service elasticsearch status
elasticsearch dead but subsys locked

Then i checked the subsys directory and found that an empty file named elasticsearch is getting generated
[claritygbl@gdclgdvaplv01d elasticsearch]$ cd /var/lock/subsys
[claritygbl@gdclgdvaplv01d subsys]$ ls -lrt
total 0
-rw-r--r--. 1 root root 0 Dec 27 08:18 lvm2-monitor
-rw-r--r--. 1 root root 0 Dec 27 08:18 network
-rw-r--r--. 1 root root 0 Dec 27 08:18 auditd
-rw-------. 1 root root 0 Dec 27 08:18 rsyslog
-rw-r--r--. 1 root root 0 Dec 27 08:18 rpcbind
-rw-r--r--. 1 root root 0 Dec 27 08:18 messagebus
-rw-r--r--. 1 root root 0 Dec 27 08:18 NetworkManager
-rw-r--r--. 1 root root 0 Dec 27 08:18 avahi-daemon
-rw-r--r--. 1 root root 0 Dec 27 08:18 rpc.idmapd
-rw-r--r--. 1 root root 0 Dec 27 08:18 netfs
-rw-r--r--. 1 root root 0 Dec 27 08:18 winbindd
-rw-r--r--. 1 root root 0 Dec 27 08:18 blk-availability
-rw-r--r--. 1 root root 0 Dec 27 08:18 cups
-rw-r--r--. 1 root root 0 Dec 27 08:18 acpid
-rw-r--r--. 1 root root 0 Dec 27 08:18 haldaemon
-rw-r--r--. 1 root root 0 Dec 27 08:18 autofs
-rw-r--r--. 1 root root 0 Dec 27 08:18 mcelogd
-rw-r--r--. 1 root root 0 Dec 27 08:18 sshd
-rw-r--r--. 1 root root 0 Dec 27 08:18 xinetd
-rw-r--r--. 1 root root 0 Dec 27 08:18 ntpd
-rw-r--r--. 1 root root 0 Dec 27 08:18 vsftpd
-rw-r--r--. 1 root root 0 Dec 27 08:18 abrt-ccpp
-rw-r--r--. 1 root root 0 Dec 27 08:18 abrtd
-rw-r--r--. 1 root root 0 Dec 27 08:18 abrt-oops
-rw-r--r--. 1 root root 0 Dec 27 08:18 qpidd
-rw-r--r--. 1 root root 0 Dec 27 08:18 crond
-rw-r--r--. 1 root root 0 Dec 27 08:18 atd
-rw-r--r--. 1 root root 0 Dec 27 08:18 networker
-rw-r--r--. 1 root root 0 Dec 27 08:18 rhnsd
-rw-r-----. 1 root root 0 Dec 27 08:18 rhsmcertd
-rw-r--r--. 1 root root 0 Dec 27 08:18 oddjobd
-rw-r--r--. 1 root root 0 Dec 27 08:19 libvirt-guests
-rw-r--r--. 1 root root 0 Dec 27 08:19 local
-rw-r--r--. 1 root root 0 Jan 18 08:15 postfix
-rw-r--r--. 1 root root 0 Jan 19 11:18 elasticsearch

The log file details are :
[2017-01-19T11:14:10,201][INFO ][o.e.n.Node ] [clar-1] initializing ...
[2017-01-19T11:14:10,343][INFO ][o.e.e.NodeEnvironment ] [clar-1] using [1] data paths, mounts [[/var (/dev/mapper/rootvg-varlv)]], net usable_space [6.5gb], net total_space [8.4gb], spins? [possibly], types [ext4]
[2017-01-19T11:14:10,344][INFO ][o.e.e.NodeEnvironment ] [clar-1] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-01-19T11:14:10,347][INFO ][o.e.n.Node ] [clar-1] node name [clar-1], node ID [4qTcv-q4QHWcY3igjhERrQ]
[2017-01-19T11:14:10,350][INFO ][o.e.n.Node ] [clar-1] version[5.1.1], pid[3384], build[5395e21/2016-12-06T12:36:15.409Z], OS[Linux/2.6.32-642.11.1.el6.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_40/25.40-b25]
[2017-01-19T11:14:11,621][INFO ][o.e.p.PluginsService ] [clar-1] loaded module [aggs-matrix-stats]
[2017-01-19T11:14:11,621][INFO ][o.e.p.PluginsService ] [clar-1] loaded module [ingest-common]
[2017-01-19T11:14:11,621][INFO ][o.e.p.PluginsService ] [clar-1] loaded module [lang-expression]
[2017-01-19T11:14:11,621][INFO ][o.e.p.PluginsService ] [clar-1] loaded module [lang-groovy]
[2017-01-19T11:14:11,621][INFO ][o.e.p.PluginsService ] [clar-1] loaded module [lang-mustache]
[2017-01-19T11:14:11,621][INFO ][o.e.p.PluginsService ] [clar-1] loaded module [lang-painless]
[2017-01-19T11:14:11,621][INFO ][o.e.p.PluginsService ] [clar-1] loaded module [percolator]
[2017-01-19T11:14:11,621][INFO ][o.e.p.PluginsService ] [clar-1] loaded module [reindex]
[2017-01-19T11:14:11,621][INFO ][o.e.p.PluginsService ] [clar-1] loaded module [transport-netty3]
[2017-01-19T11:14:11,622][INFO ][o.e.p.PluginsService ] [clar-1] loaded module [transport-netty4]
[2017-01-19T11:14:11,622][INFO ][o.e.p.PluginsService ] [clar-1] no plugins loaded
[2017-01-19T11:14:14,820][INFO ][o.e.n.Node ] [clar-1] initialized
[2017-01-19T11:14:14,820][INFO ][o.e.n.Node ] [clar-1] starting ...
[2017-01-19T11:14:15,040][INFO ][o.e.t.TransportService ] [clar-1] publish_address {10.247.132.109:9300}, bound_addresses {10.247.132.109:9300}
[2017-01-19T11:14:15,046][INFO ][o.e.b.BootstrapCheck ] [clar-1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-01-19T11:14:15,050][ERROR][o.e.b.Bootstrap ] [clar-1] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2017-01-19T11:14:15,060][INFO ][o.e.n.Node ] [clar-1] stopping ...
[2017-01-19T11:14:15,121][INFO ][o.e.n.Node ] [clar-1] stopped
[2017-01-19T11:14:15,122][INFO ][o.e.n.Node ] [clar-1] closing ...
[2017-01-19T11:14:15,143][INFO ][o.e.n.Node ] [clar-1] closed

what to do now, why is this empty file getting generated, i have no idea. kindly help me out. Thanks in advance

jasontedor · January 19, 2017, 12:16pm

Please check in /etc/security/limits.d/ for the existence of a file that is overriding your limit (grep nproc /etc/security/limits.d/*); I bet there is a file matching *-nproc.conf overriding your limit.

bini · January 19, 2017, 12:50pm

Yes, there was a * - nproc.conf file in /etc/security/limits.d/ afterall !!
Finally i am able to run elasticsearch after changing the nproc value . Thanks a ton @jasontedor for your great help.

jasontedor · January 19, 2017, 2:31pm

You're very welcome.

system · February 16, 2017, 2:31pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.