However, sometimes I found that the logstash will keep opening the rotated file such as /tmp/data/output.log-20220201 when I am checking in the lsof | grep "deleted". The disk could not be freed because of this. Currently, the only way to free the disk is to kill the logstash process and restart it. This never happened before.
and in the output section sprintf the path option in the file output
file { path => "%{[@metadata][filename]}" ... }
You will always have a file descriptor open to /dev/null, but /home/user/data/file will get closed. The test of whether a file is active is done immediately after events are processed, so if the path option is a constant the file will never be inactive.
Note that there is an open issue for this and the code changes needed are trivial.
which will still take some spaces in the disk (around 1k per message).
As it is stated, " If you use an absolute path you cannot start with a dynamic string. E.g: /%{myfield}/ , /test-%{myfield}/ are not valid paths". Thus, when the path is a variable/dynamic, the Logstash won't use the absolute path.
Yeah, if @path contains a sprintf reference it is expanded relative to the working directory. If you try to add a / before the sprintf reference then you get an exception. The following works if you ln -s /dev/null /tmp/null
However, this issue tells me that when java_execution is enabled, an empty batch is sent through the pipeline every 15 seconds, so multi_receive_encoded is called to process a batch that contains no events. That's enough to get close_stale_files called.
So you can get the behaviour you want by enabling the Java execution engine, which is the only choice you have when 8.0 arrives.
The holding spaces were immediately released after config file changed as it is mentioned and pipeline reloaded. I am not sure whether it is worked by reloading the pipeline or heartbeat events. So I will continue to monitor the disk usage to see if the issue is resolved. Thanks.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.