Preventing the Logstash Log File from Filling Up Disk

Elastic Stack Logstash
1

Hello All,

Recently I had an issue on one of my ELK nodes where it would not start for some reason. The issue was investigated further and it turns out that the Logstash log file was eating up all of the disk space for the host. We had to delete this log file to free up space, then the node was able to be brought back to life.

How do you keep this from happening? Is there documentation on this as well?

Thanks

2

With the default configuration Logstash hardly logs anything so the first step would be to reduce the logging. You might also want to make sure you have log rotation in place. I think the Logstash Debian/RPM packages contain configuration for that.

3

Could you please elaborate on "reduce the logging". I stated that is what I was trying to do.

4

Why was it filling up? There must have been an error causing it. Stopping that error would be the best way.

5

Could you please elaborate on "reduce the logging". I stated that is what I was trying to do.

Since Logstash barely logs anything by default, there are basically two reasons why you might have large logs:

  • You have enabled extra verbose logs with --verbose or --debug, or
  • you have tons of errors on your logs.

So, check the level of logging and make sure you're not logging tons of errors. Also, make sure you don't have a stdout output in your configuration that's writing all events passing through Logstash to the log directory.