This suggests to me that the Elasticsearch user does not have write privileges to /var/log/back. Make sure the user running the elasticsearch process (ostensibly elasticsearch, since you installed the RPM package) has read/write privileges to this path on all data and master nodes.
1 Like
Elasticsearch user? How can I check it?
ls -l /var/log | grep back
The user and group owner of the mount point will be listed by this command.
Thank you:
[root@localhost ~]# ls -l /var/log | grep back
drwxr-xr-x. 2 root root 6 Aug 27 12:25 back
drwxr-xr-x. 2 root root 6 Aug 27 12:27 back-longIt's clear from this that the elasticsearch user will not have write privileges to this folder.
On my own systems:
$ grep elastic /etc/passwd
elasticsearch:*:700:700:Elasticsearch User:#other stuff here...
I made my elasticsearch user with UID and GID of 700. Yours may differ.
On my server:
path.repo: ['/bigdisk/repos']
$ ls -ln /bigdisk
drwxr-xr-x 5 700 700 5 Sep 16 2016 repos
This same elasticsearch user must have the same UID at least (and preferably GID) on every master node and data node. Also, this user must have read/write privileges to the path on the data node. My mount point is /bigdisk, and /bigdisk/repos is a subdirectory owned by UID/GID 700, so that the elasticsearch user can read from and write to that path.
Mine is:
# grep elastic /etc/passwd
elastic:x:1000:1000:elastic:/home/elastic:/bin/bash
elasticsearch:x:996:994:elasticsearch user:/home/elasticsearch:/sbin/nologinand :
# ls -ln back
drwxr-xr-x. 2 0 0 6 Aug 27 12:25 back
drwxr-xr-x. 2 0 0 6 Aug 27 12:27 back-longand I did:
# chown -R elastic:elastic back
# chown -R elastic:elastic back-long/
# ls -ln
drwxr-xr-x. 2 1000 1000 6 Aug 27 12:25 back
drwxr-xr-x. 2 1000 1000 6 Aug 27 12:27 back-longBut, I got same error. Why?
I see two "elastic" group:
# cat /etc/group | grep elastic
wheel:x:10:elastic
elastic:x:1000:elastic
elasticsearch:x:994:Which one is OK?
In all honesty, I don't know. The output from ps auwwx | grep elast should show you which user is running Elasticsearch.
I presume it should be the elasticsearch user, as it has a UID less than 1000, which implies to me that this is the one that is auto-generated when you install the RPM.
The next point is that the mount point isn't the important one to have UID/GID set, but the directory underneath that. I suggest you make your path.repo a subdirectory of the mount points /var/log/back and /var/log/back-long, e.g. /var/log/back/repo. Then make the repo dir owned by the elasticsearch user. The mount-point itself will almost always be reset to be owned by root, so use a subdirectory.
I did:
# ps auwwx | grep elast
elastic+ 3628 0.3 33.9 8203572 2691240 ? Ssl Aug27 122:09 /bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch
root 21760 0.0 0.0 112648 964 pts/0 S+ 13:18 0:00 grep --color=auto elastand
# chown -R elasticsearch:elasticsearch back-long/
# chown -R elasticsearch:elasticsearch back
# ls -l
d---------. 2 elasticsearch elasticsearch 6 Aug 27 12:25 back
drwxr-xr-x. 2 elasticsearch elasticsearch 6 Aug 27 12:27 back-longThen created a "repo" directory inside of directories:
# ls -l
total 0
drwxr-xr-x. 2 elasticsearch elasticsearch 6 Sep 20 13:31 repoThen, Change the config file as below:
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
path.repo: ["/var/log/back/repo","/var/log/back-long/repo"]
#
# Path to log files:
#
path.logs: /var/log/elastic
#Then, I did:
# service elasticsearch restart
Restarting elasticsearch (via systemctl): [ OK ]
# service elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Wed 2017-09-20 13:33:58 +0430; 2min 24s ago
Docs: http://www.elastic.co
Process: 21992 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 21989 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 21992 (code=exited, status=1/FAILURE)
Sep 20 13:33:58 localhost.localdomain elasticsearch[21992]: 2017-09-20 13:33:58,484 main ERROR Null object returned for RollingFile in Appenders.
Sep 20 13:33:58 localhost.localdomain elasticsearch[21992]: 2017-09-20 13:33:58,484 main ERROR Null object returned for RollingFile in Appenders.
Sep 20 13:33:58 localhost.localdomain elasticsearch[21992]: 2017-09-20 13:33:58,484 main ERROR Null object returned for RollingFile in Appenders.
Sep 20 13:33:58 localhost.localdomain elasticsearch[21992]: 2017-09-20 13:33:58,485 main ERROR Unable to locate appender "rolling" for logger config "root"
Sep 20 13:33:58 localhost.localdomain elasticsearch[21992]: 2017-09-20 13:33:58,485 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger co...og.index"
Sep 20 13:33:58 localhost.localdomain elasticsearch[21992]: 2017-09-20 13:33:58,485 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger conf....slowlog"
Sep 20 13:33:58 localhost.localdomain elasticsearch[21992]: 2017-09-20 13:33:58,485 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.e...recation"
Sep 20 13:33:58 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Sep 20 13:33:58 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Sep 20 13:33:58 localhost.localdomain systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.Why I got error?
Those errors are unrelated, I think, unless you also changed /var/log/elastic, which directory must also be owned by the elasticsearch user and group. The error you are showing there is related to Elasticsearch's logging output.
I never did any permission on it:
$ ls -l
drwxr-xr-x. 2 root root 6 Jul 21 17:03 elastic
drwxr-x---. 2 elasticsearch elasticsearch 4096 Jul 21 17:04 elasticsearchAs you see "elasticsearch" can't run 
So, why not set path.logs to /var/log/elasticsearch then, since it's already owned by the elasticsearch user/group.
Is it my problem? OK, I did it but not matter:
[root@localhost ~]# service elasticsearch restart
Restarting elasticsearch (via systemctl): [ OK ]
[root@localhost ~]# service elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sun 2017-09-24 09:53:16 +0330; 2s ago
Docs: http://www.elastic.co
Process: 25156 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 25153 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 25156 (code=exited, status=1/FAILURE)
Sep 24 09:53:15 localhost.localdomain systemd[1]: Starting Elasticsearch...
Sep 24 09:53:15 localhost.localdomain systemd[1]: Started Elasticsearch.
Sep 24 09:53:16 localhost.localdomain systemd[1]: elasticsearch.service: main...
Sep 24 09:53:16 localhost.localdomain systemd[1]: Unit elasticsearch.service ...
Sep 24 09:53:16 localhost.localdomain systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.Check the logs for the new reason it's not starting.
The log is:
# cat elasticsearch.log
[2017-09-24T09:53:16,363][ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalStateException: Unable to access 'path.repo' (/var/log/back/repo)
at org.elasticsearch.bootstrap.Security.addPath(Security.java:413) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:296) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:224) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Security.configure(Security.java:114) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:350) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.1.jar:5.4.1]
Caused by: java.nio.file.AccessDeniedException: /var/log/back/repo
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_131]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_131]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_131]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_131]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_131]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_131]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_131]
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:455) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Security.addPath(Security.java:411) ~[elasticsearch-5.4.1.jar:5.4.1]
... 12 more
[2017-09-24T09:53:16,372][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.repo' (/var/log/back/repo)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.4.1.jar:5.4.1]
Caused by: java.lang.IllegalStateException: Unable to access 'path.repo' (/var/log/back/repo)
at org.elasticsearch.bootstrap.Security.addPath(Security.java:413) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:296) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:224) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Security.configure(Security.java:114) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:350) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.1.jar:5.4.1]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /var/log/back/repo
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_131]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_131]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_131]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_131]
at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_131]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_131]
at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_131]
at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:455) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Security.addPath(Security.java:411) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:296) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:224) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Security.configure(Security.java:114) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:350) ~[elasticsearch-5.4.1.jar:5.4.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.1.jar:5.4.1]
... 6 moreAny idea? I guess the permission is a problem!!!
Seems pretty clear to me. For whatever reason, permissions notwithstanding, Elasticsearch is telling you that it cannot write to (and possibly read from) this directory.
Can you (as root) do su - elasticsearch? If you can do this, what happens when you then cd /var/log/back and try an ls? And then cd /var/log/back/repo and do an ls? and then try touch test_file.txt? If the elasticsearch user can do this on all master and data nodes in your cluster, then this error should not occur.
No, I can't:
# su - elasticsearch
su: warning: cannot change directory to /home/elasticsearch: No such file or directory
This account is currently not available.and:
# cd /home/
# ls
elastic"elasticsearch" not exist.
Why? Thus, The user must be "elastic" not "elasticsearch" and if you remember I changed the owner of directory to "elastic" but got same error!!!
I did:
# su - elastic
[elastic@localhost ~]$ cd /var/log/back
-bash: cd: /var/log/back: Permission deniedbut:
[elastic@localhost ~]$ cd /var/log/back-long/
[elastic@localhost back-long]$ ls
repoIt is odd!!!
$ ls -l
d---------. 3 elasticsearch elasticsearch 17 Sep 20 13:31 back
drwxr-xr-x. 3 elasticsearch elasticsearch 17 Sep 20 13:31 back-longWhat is your idea?
@theuntergeek I guess permission is not a problem because I solved it via "chmod" command:
$ ls -l
drwxr-xr-x. 3 elastic elastic 17 Sep 20 13:31 back
drwxr-xr-x. 3 elastic elastic 17 Oct 2 12:30 back-longand
[elastic@localhost back]$ ls -l
total 0
drwxr-xr-x. 2 elastic elastic 6 Sep 20 13:31 repo
[elastic@localhost back]$ cd repo/
[elastic@localhost repo]$ mkdir test
[elastic@localhost repo]$ ls
testBut when I restart "elasticsearch" service then it failed. In "Elasticsearch" config file when I add "/var/log/back/repo" then elasticsearch failed!!!
When my config is:
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
path.repo: ["/var/log/back-long/repo"]
#
# Path to log files:
#
path.logs: /var/log/elastic
#path.logs: /var/log/elasticsearch
#then Elasticsearch is OK and below code :
PUT /_snapshot/my_backup
{
"type": "fs",
"settings": {
"compress": true,
"location": "/var/log/back-long/repo"
}
}Give me:
{
"statusCode": 502,
"error": "Bad Gateway",
"message": "connect ECONNREFUSED 127.0.0.1:9200"
}Why?
That's an interesting result. What IP are you connecting to Kibana with? Is it 127.0.0.1? Or is it something else? Do you have a firewall on port 9200 that is allowing GET commands but not PUT or POST commands?
Hey, I changed the owner of "/var/log/elastic" directory to "elastic" and it seems OK.
I got same error:
{
"error": {
"root_cause": [
{
"type": "exception",
"reason": "failed to create blob container"
}
],
"type": "exception",
"reason": "failed to create blob container",
"caused_by": {
"type": "access_denied_exception",
"reason": "/var/log/back-long/repo/tests-ETLXOhbVSSyaDAcasP0fwQ"
}
},
"status": 500
}Why?
I did:
[root@localhost log]# chmod -R 777 back-long/
[root@localhost log]# chmod -R 777 back
[root@localhost log]# ls -l
drwxrwxrwx. 3 elastic elastic 17 Oct 2 12:38 back
drwxrwxrwx. 3 elastic elastic 17 Oct 2 12:30 back-longand code give me:
{
"acknowledged": true
}What is "elasticsearch" user and group?
elastic user and group are not correct. The process for Elasticsearch is owned by the user and group elasticsearch.