Logstash keystore Permisson denied

Elastic Stack Logstash
1

We have configured smb shared folder between elastic stack servers, that contain the logstash.keystore file.

This storage is mounted in for example /mnt/elastic folder

the file /mnt/elastic/logstash.keystore is symlinked into /etc/logstash/logstash.keystore and for sure also into /usr/share/logstash/config/logstash.keystore.

The environment variables are setuped.

When i run following command, It works well:

./bin/logstash-keystore list

When I try to remove not existing key, it also properly detect, that key does not exists

But when I try to add a new key, it fail for following error:

ERROR] 2020-01-02 15:58:33.245 [main] secretstorecli - Error while trying to store secret urn:logstash:secret:v1:new_key {:cause=>java.io.IOException: Permission denied, :backtrace=>["org.logstash.secret.store.backend.JavaKeyStore.persistSecret(org/logstash/secret/store/backend/JavaKeyStore.java:318)", "org.logstash.secret.cli.SecretStoreCli.add(org/logstash/secret/cli/SecretStoreCli.java:169)", "org.logstash.secret.cli.SecretStoreCli.command(org/logstash/secret/cli/SecretStoreCli.java:104)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:425)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:292)", "RUBY.<class:SecretStoreCli>(/usr/share/logstash/lib/secretstore/cli.rb:35)", "RUBY.<main>(/usr/share/logstash/lib/secretstore/cli.rb:16)", "org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:889)", "org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:893)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:782)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:795)", "org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:607)", "org.jruby.Main.doRunFromMain(org/jruby/Main.java:412)", "org.jruby.Main.internalRun(org/jruby/Main.java:304)", "org.jruby.Main.run(org/jruby/Main.java:234)", "org.jruby.Main.main(org/jruby/Main.java:206)"]}

I have tried almost everything:
User: logstash or elastic
Group: elastic
mask: 600, 660, 666

What can I do more?

2

on the logstash server are you able to

sudo -Hu logstash touch /mnt/elastic/delete_me ?

3

Yes :slight_smile:

4

When I temporarly move the original file elsewhere and tried to create a new keystore file, it fail on the following error

[ERROR] 2020-01-06 07:58:30.124 [main] secretstorecli - Error while trying to create the Logstash keystore.  {:cause=>java.nio.file.FileAlreadyExistsException:
/usr/share/logstash/config/logstash.keystore, :backtrace=>["org.logstash.secret.store.backend.JavaKeyStore.create(org/logstash/secret/store/backend/JavaKeyStore.java:101)", "org.logstash.secret.store.backend.JavaKeyStore.create(org/logstash/secret/store/backend/JavaKeyStore.java:40)", "org.logstash.secret.store.SecretStoreFactory.doIt(org/logstash/secret/store/SecretStoreFactory.java:111)", "org.logstash.secret.store.SecretStoreFactory.create(org/logstash/secret/store/SecretStoreFactory.java:75)", "org.logstash.secret.cli.SecretStoreCli.deleteThenCreate(org/logstash/secret/cli/SecretStoreCli.java:188)", "org.logstash.secret.cli.SecretStoreCli.create(org/logstash/secret/cli/SecretStoreCli.java:182)", "org.logstash.secret.cli.SecretStoreCli.command(org/logstash/secret/cli/SecretStoreCli.java:67)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:425)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:292)", "RUBY.<class:SecretStoreCli>(/usr/share/logstash/lib/secretstore/cli.rb:35)", "RUBY.<main>(/usr/share/logstash/lib/secretstore/cli.rb:16)", "org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:889)",
"org.jruby.Ruby.runInterpreter(org/jruby/Ruby.java:893)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:782)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:795)", "org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:607)", "org.jruby.Main.doRunFromMain(org/jruby/Main.java:412)", "org.jruby.Main.internalRun(org/jruby/Main.java:304)", "org.jruby.Main.run(org/jruby/Main.java:234)", "org.jruby.Main.main(org/jruby/Main.java:206)"]}

It seems like the logstash is not able to work with symlinks..

5

When I remove the symlink and move the original file to the destination with exactly same user/group/rights, it's working well

6

So the temporary hotfix is

rm /usr/share/logstash/config/logstash.keystore
cp /mnt/elastic/logstash.keystore /usr/share/logstash/config/logstash.keystore
 
# Manipulation with the .keystore file
 
mv -f /mnt/elastic/logstash.keystore /mnt/elastic/logstash.keystore.old
cp /usr/share/logstash/config/logstash.keystore /mnt/elastic/logstash.keystore # When its done by mv comand, it failed on "mv: preserving times for '/mnt/elastic_ks/logstash.keystore': Operation not permitted" :)
rm /usr/share/logstash/config/logstash.keystore
ln -s /mnt/elastic/logstash.keystore /usr/share/logstash/config/logstash.keystore
 
# Check thru ./bin/logstash-keystore list that everything works well
 
rm -f /mnt/elastic/logstash.keystore.old
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.