I'm working to secure all my Elastic communications channels and config files. I use the elastic and kibana keystore utilities to store all the passwords that are scattered throughout the config files. This now works and all my kibana and ES config files are rid of plaintext passwords and all communications are encrypted.
Now I'm moving to Logstash. I'm running ES v6.4 on all nodes using the CentOS 7 rpm repository version. As a first step, I want to set up the logstash keystore to keep all the passwords. I created the keystore password environmental config as follows:
vi /etc/sysconfig/logstash.
Add line.. LOGSTASH_KEYSTORE_PASS="mypassword"
chown root:root /etc/sysconfig/logstash
chmod 600 /etc/sysconfig/logstash
I assume the previous sysconfig setting is for the logstash service once it starts. I load the environment variable into my shell (not running as root):
set +o history
export LOGSTASH_KEYSTORE_PASS="mypassword"
set -o history
Next I create the keystore in the /etc/logstash directory:
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create
This generates the following output, which seems to have completed correctly but it created a strange directory that is related to the error messages, as shown below:
2018-11-21 10:44:57,996 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2018-11-21 10:44:57,997 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2018-11-21 10:44:57,998 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2018-11-21 10:44:57,998 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Created Logstash keystore at /etc/logstash/logstash.keystore
ls-node-1$ ls -altr /etc/logstash
....
drwxr-xr-x. 2 root root 96 Nov 21 10:44 ${sys:ls.logs}
-rw-r--r--. 1 root root 383 Nov 21 10:45 logstash.keystore
....
I press on in spite of the errors. Next I add a password for the xpack.monitoring.elasticsearch.password entry:
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add MON_ES_PWD
2018-11-21 10:54:14,119 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2018-11-21 10:54:14,120 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2018-11-21 10:54:14,120 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2018-11-21 10:54:14,121 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Enter value for MON_ES_PWD:
Added 'mon_es_pwd' to the Logstash keystore.
Again, the errors, however, the entry seems to be in place.
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list
2018-11-21 11:01:30,100 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2018-11-21 11:01:30,101 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2018-11-21 11:01:30,101 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2018-11-21 11:01:30,102 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
mon_es_pwd
I'm not confident that this is working correctly given the symptoms above. I hesitate to blindly enter the MON_ES_PWD variable in my logstash config and bring down my ingest pipeline in the process. Any advice?