APT Package: Queue directory not writeable

Elastic Stack Logstash
1

Has anyone else experienced the issue when installing Logstash 5.x as an APT-Package, that it won't start:

[2016-12-19T07:06:27,418][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:420:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:202:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:118:in `validate_all'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:117:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:205:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:178:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

It seems like directory is being created as root - instead of user logstash, and thus is not writeable.
This happened to us on two machines already that we installed Logstash 5 on.

....:/var/log/logstash$ ls -al /usr/share/logstash/data/queue
total 8
drwxr-xr-x 2 root     root     4096 Dec 19 06:48 .
drwxrwxr-x 3 logstash logstash 4096 Dec 19 06:48 ..

After chown-ing the directory to the logstash user, it works again.

Is this an issue on my part or shall I open an issue on GitHub?

Also: When requesting the validation of the configuration in this state (-t / --config.test_and_exit), it does not terminate or provide the output above!

Edit: Changed to pre-formatted text, include the error message properly

2

Always format log snippets as preformatted text. If you read what you've posted clearly you'll find that the actual error message is missing.

Also: When requesting the validation of the configuration in this state (-t / --config.test_and_exit), it does not terminate or provide the output above!

You should not expect -t to find all problems, especially not problems like this where the configuration files are okay and its the environment that's broken.

3

Thank you for your feedback.

I reformatted the message and included the error - sorry about that.
(Interestingly, when editing it and using the "preformatted text"-button in the editor, it used backticks again instead of using the four spaces.

4

The debian package does not create this folder, just checked with
dpkg -L logstash | grep queue
on Ubuntu 14.04 and 16.04.

The dir was only created after enabling the persistent queue, but with correct permissions.
Did you run logstash as root for testing something, when you first enabled the persistent queue?

5

Thanks for your analysis.

This is quite likely - in our setup, we install the plugins for logstash and then perform the configuration check as root (both via ansible):

sudo /usr/share/logstash/bin/logstash-plugin install logstash-filter-de_dot
sudo /usr/share/logstash/bin/logstash -t --path.settings /etc/logstash

Both would run as root, and if the check creates this folder as root, we might indeed run into the problem when starting logstash later as a normal service...

I'll try to reproduce this and report back if it's another issue.
Thanks for the assistance