I'm running Logstash 7.17.0 on my Windows server to ship data into my Elasticsearch 8.11.0 instance, running on a Linux server. The Logstash instance is set to ingest log files from a folder, which is actively being written to by another application. These files are eventually rotated by the application (e.g., file1.log is renamed into file1.OLD, and a new file1.log is created)
Most of the times, Logstash's ingests the files as expected, whilst allowing the other application to write new log lines to said files. However, during the file rotation, Logstash seems to lock/hold onto the files, which results in a denied access error by the other application.
Most times, this behavior resolves after some time, but occasionally, Logstash needs to be restarted to let go of the file lock.
Has this problem appeared with this version of Logstash before? Moreover, is there any indication on the possible cause and solution?
We have .log files, which logstash reads as its input. These files are created/written to by an application. After x number of lines or x number of days, the files are rotated.
For instance, my_file.log will be renamed into my_file.OLD (only the extension is changed, the file name remains the same). And again the application will create a new my_file.log to record new events.
The expected behaviour was for logstash to start monitoring the new file as soon as my_file is rotated into .OLD file. In our File input plugin, we also explicitly set logstash to ignore any files with .OLD extension to ensure this.
logstash cannot open the newly rotated log file until after rotation has occurred. The new log file will not exist, and logstash will not access it, if there is a file access error when creating it. If the application has successfully created the log file then how can it get an access error? You need to provide more detail.
Apologies, I think I was being unclear. The application itself throws an 'access denied' error when trying to rename the old log file. This error is fixed only upon restarting the Logstash instance, which then releases the file lock.
Only then, do we see a new log file created. The problem, therefore, was not Logstash not having access to the file, but rather, Logstash causing the files to not be accessible by other applications (i.e., for file operations, such as renaming).
We tried adjusting some parameters in the File input plugin (e.g., close_older, stat_file) to reduce the times Logstash checks and presumably hold onto the log files. While this does seem to reduce the 'access denied' error by the application's renaming/rotating process, this implies that some files are rotated before Logstash could do a complete scan of the files, resulting in an incomplete log ingestion process.
Perhaps, there are some behaviours of Logstash that could've caused this? I'm unsure if Logstash has some adjustable file handler settings like Filebeat here:
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.