Yesterday I started using a config that utilizes the elasticsearch input to run a query. However I noticed the following rather insane behavior:
All the output received from elastic after Logstash runs the query is saved to /var/log/syslog. This created multi GB files, making my server run out of disk space and crashing it.
As all the queries are saved to syslog, its all plain text and there for everybody to see.
How can I stop this insanity? I absolutely cannot think of a single reason why this needs to be saved to syslog.
PS. Also noticed the DNS lookup filter also outputs failed lookups to syslog. Again, not necessary.
Edit: It is not only the elastic input, for whatever reason now Logstash is outputting everything to syslog! WTF. I made no changes to Logstash or the existing configs, I removed the two new configs but still same thing is happening.
Running a plain Logstash install on a plain Ubuntu server.
One rogue config that had a output misconfigured. But this misconfiguration has been present for quite some time so that was not the root cause.
No, I found out that again the elastic input plugin refuses to correctly parse queries (this query works perfectly using curl or dev tools in Kibana). I have range filter in my query which Logstash decided it didn't care about so instead it queried the complete index and dumped all the output in syslog. Brilliant.
So I converted my query and config to pipe output from curl to a http input. This worked fine for one config. But adding a second http input, on a different port in a different config, just broke the whole thing. Even removing one the configs doesn't fixes the problem. CPU now constantly pegged at 100% and starting Logstash takes forever and it doesn't appear to reach the point where it actually starts doing something. The log as usual don't provide any useful information when you need it.
logstash does not write to syslog. However if you are running it as a service then the service manager may cc its stdout to syslog. You can turn that off in the service manager (systemd or whatever your system uses).
If logstash startup time has become unreasonably large verify that there are not extra files in path.config other than the configuration files you want it to read. It will read in anything in the directory if you just point it to a directory.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.