Getting permission denied error after changing data directory for logstash

Elastic Stack Logstash
1

Hi Team,

I am getting permission denied error after changing path.data and path.logs in logstash to new path.

Same issue is happening for kibana.

However for Elasticsearch it is working, i can see elasticsearch folder inside the new es path's lib and log directory.

Bydefault, it is creating folders inside /var/lib and /var/log, and there is no issue if i keep this path.

I want to change default location to new path so i have edited each component's config file and mentioned new path for path.data and `path.logs.

Below are the permission details -

/var, /var/lib and /var/log is owned by root and having 755 and i can see all the folders like kibana, logstahsh, Elasticsearch are getting created properly.

New path is /new,

/new, /new/lib and /new/log is also owned by root and having 755 but here only elastichsearch is able to create folder and not logstash or kibana.

I dont want to give/change permission and ownership manually after the cluster installtion, this should happen properly when cluster is getting installed.

I will upload error logs soon.

Due to this issue logstash and kibana services are not getting started.

Can someone point where can be the issue?

Thanks,

2

logstash logs

[FATAL] [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/new/lib/logstash" does not exist, and I failed trying to create it: Errno::EACCES - Permission denied - /new/lib/logstash>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:614:in `block in value'", "org/jruby/RubyKernel.java:1891:in `tap'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:606:in `value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:135:in `get_value'", "/usr/share/logstash/logstash-core/lib/logstash/environment.rb:122:in `block in LogStash'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:196:in `block in post_process'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:195:in `post_process'", "/usr/share/logstash/logstash-core/lib/logstash/util/settings_helper.rb:43:in `post_process'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:295:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:291:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:93:in `<main>'"]}
[FATAL] [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
 logstash.service: main process exited, code=exited, status=1/FAILURE
 Unit logstash.service entered failed state.
 logstash.service failed.
 logstash.service holdoff time over, scheduling restart.
 Stopped logstash.
 Started logstash.
 logstash[23079]: Using bundled JDK: /usr/share/logstash/jdk
 logstash[23079]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

kibana logs

kibana[21416]: FATAL  Error: EACCES: permission denied, mkdir '/new/lib/kibana'
 kibana.service: main process exited, code=exited, status=1/FAILURE
 Unit kibana.service entered failed state.
 kibana.service failed.
 kibana.service holdoff time over, scheduling restart.
 Stopped Kibana.
3

What user to the kibana and logstash services run as?

4

@Badger,

Although kibana service is not running but it will run with kibana user.

and logstash service is getting restarted again and again so checked the process and it is running with logstash user.

5

What are the permissions for these directories? Logstash needs to have write permissions to the directories in path.data and path.logs, the same applies to the other services.

6

installation got stopped due to this issue.

it is still creating all directories in /var/lib path as well.

[root@ lib]# ls -ld elasticsearch logstash kibana/

drwxr-s---. 2 elasticsearch elasticsearch 6 Dec 18 19:49 elasticsearch
drwxr-s---. 2 kibana        kibana        6 Dec 18 20:24 kibana/
drwxr-xr-x. 2 logstash      logstash      6 Dec 18 20:01 logstash

[root@ lib]# pwd
/var/lib

[root lib]# ls -ld .
drwxr-xr-x. 30 root root 4096  17:11 .
[root@ lib]#

new path -

[root@ lib]# ls -ld /new/*
drwxr-xr-x. 3 root root   17:06 /new/lib
drwxr-xr-x. 3 root root   17:06 /new/log
[root@ lib]#

only elasticsearch directory is created in new path.

[root@ lib]# ls -ld /new/lib/*
drwxr-s---. 3 elasticsearch elasticsearch 19  12:08 /new/lib/elasticsearch
[root@ lib]#

[root@ lib]# ls -ld /new/log/*
drwxr-s---. 2 elasticsearch elasticsearch 4096  12:08 /new/log/elasticsearch
[root@ lib]#

Below is logstash.yml file -

path.data: /new/lib/logstash
path.logs: /new/log/logstash

kibana.yml -

path.data: /new/lib/kibana

elasticsearch.yml

path.data: /new/lib/elasticsearch
path.logs: /new/log/elasticsearch

Thanks,

7

does /new/lib/logstash and /new/log/logstash needs to be create first and give ownership of logstash user? this way it may work, but don't think this is the correct way.

for elasticsearch, i have not created elasticsearch directory in /new/lib and /new/log first manually but still this is getting created in new path.

Thanks,

8

How are you installing?

If I'm not wrong, installing using the packages will create the directories under the /var path and set the correct permissions, you can check the directory layout here.

If you are planning to use a different path, you need to make sure that the directories you are going to use exists and have the correct permissions.

So, you will need to create /new/lib/logstash and /new/log/logstash and set the write permissions to the logstash user.

9

Yes installing via .rpm packages.

I am installing cluster from scratch so before starting i cannot give ownership of logstash or kibana user to their directories.

Thanks,

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.