Hi!
I've the same issue... I think this is not related to logstash.yml or any file in conf.d. It has something todo with log4j2.
When I test my config it says:
$ /usr/share/logstash/bin/logstash -f /etc/logstash --config.test_and_exit
2017-09-08 17:10:42,115 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path //usr/share/logstash/config/log4j2.properties. Using default config which logs to console
17:10:43.255 [LogStash::Runner] FATAL logstash.runner - The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration
By adding the parameter --path.settings I could resolve the warning about log4j2:
$ /usr/share/logstash/bin/logstash -f /etc/logstash --path.settings /etc/logstash --config.test_and_exit
2017-09-08 17:31:14,808 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
But still the strange folder in /etc/logstash does not disappear:
[root@es logstash]# ll /etc/logstash
total 20
drwxrwxr-x. 2 root root 95 Sep 8 17:17 conf.d
-rw-r--r--. 1 root root 1738 Aug 14 15:12 jvm.options
-rw-r--r--. 1 root root 1334 Sep 8 16:02 log4j2.properties
-rw-r--r--. 1 root root 5660 Sep 1 14:49 logstash.yml
-rw-r--r--. 1 root root 1659 Aug 14 15:12 startup.options
drwxr-xr-x. 2 root root 46 Sep 8 17:34 ${sys:ls.logs}
EDIT:
I followed https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html to test a few things when I ran into these issues. Running logstash as service with systemctl just worked fine. So I tried a few things.
Here is what I changed:
- You cannot pass the /etc/logstash/logstash.yml config file. If you want to load multiple pipeline config files just wildcard them. I wanted to run all, so I did not add any information after the "/" (-f /etc/logstash/conf.d/)
.
[root@es bin]# pwd
/usr/share/logstash/bin
[root@es bin]# ./logstash -f /etc/logstash/conf.d/ --config.reload.automatic
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
Yes, there are still errors but I can now test my pipelines. Logstash won't die anymore.
I hope this may help someone!