It seems to me that the default configuration of logstash is that every line that is logged to /var/log/logstash/logstash-plain.log also ends up in /var/log/messages. Is that correct? I have not changed the /etc/logstash/log4j2.properties file. This didn't used to happen (5.x and earlier, I think)
Also, if I configure filebeat to read /var/log/messages and logstash to output to stdout, I can very quickly DoS the box.
How can I stop this behaviour? I only want logstash's stdout and log messages to end up in logstash-plain.log and NOT /var/log/messages.
The default log4j2 configuration does not write to /var/log/messages. My guess is that you are running logstash as a service and systemd (or whatever service manager you use) is configured to forward its stdout to /var/log/messages. You could reconfigure systemd...
Yes, using Systemd (Centos RPM). Thanks for giving me the pointer, as I believe I now have it all figured out. Logstash's Log4j configuration appears to be send Logstash internal logs to both the console (stdout) as well as the /var/log/logstash/logstash-plain.log file.
The default behaviour of systemd is to send each service's stdout and stderr to journald (and the system logs).
I can change this behaviour by modifying the log4j config to just send logs to the "rolling" logfile or by changing the service config in systemd by putting the following lines into /etc/systemd/system/logstash.service
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.