Logstash locks the file during the file rotation in windows server

Elastic Stack Logstash
1

Below is the error from logstash-plain.logstash

[2025-05-05T05:04:45,597][WARN ][filewatch.tailmode.handlers.create][main][c1077e68a0985da91e9d9117b4ed7833cd278b44745ff7a8664043bc8322aec5] failed to open file {:path=>"C:/Program Files (x86)/Netapp/Policy/logs/app.log.1", :exception=>Java::JavaNioFile::AccessDeniedException, :message=>"C:\\Program Files (x86)\\Netapp/Policy/logs/app.log.1"}


input {
  file {
    path => "C:/Program Files (x86)/Netapp/Policy/logs/app.log*"
    type => "app"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
      charset => "ISO-8859-1"
    }
    start_position => "beginning"
    sincedb_path => "C:/setup/tools/logstash/sincedb_app"
	sincedb_clean_after => 43200
	close_older => 3600
    ignore_older => 259200 # Ignore files older than 3 days
  }
}

log rotation policy

app.log when reached 10 MB gets renamed to app.log.1. Later app.log.1 renamed to app.log.2. app.log to app.log.1 and so on.

application writes to app.log.

Please suggest.

2

Hi,

That is not a logstash or Elastic problem, when your files are rotated your system seems to change the permissions.

Be aware that ingesting rotated log files will result in duplicate logs.

You should just injest app.log as log are created, if you want an history ( eg ingesting already rotated logs while setting up app.log you should do it in a separate config )

Just check permissions on files.

3

@grumo35 thank you for helping.
app.log not being read by logstash

With below configuration app.log sometimes the inode changes are not updated. when I restart logstash manually new content is updated in sincedb_app file (new inode changes). app.log is pushed by logstash.

I use tail mode app.log is actively written. Please help.

input {
  file {
    path => "C:/Program Files (x86)/Netapp/Policy/logs/app.log"
    type => "app"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
      charset => "ISO-8859-1"
    }
    start_position => "beginning"
    sincedb_path => "C:/setup/tools/logstash/sincedb_app"
	sincedb_clean_after => 43200
	close_older => 3600
    ignore_older => 259200 # Ignore files older than 3 days
  }
}
4

Try ignore_older = 0

5

Is my understanding correct. Will this solve inode recycle and no lock on archive files and also no data lost log rotation.

input {
  file {
    path => "C:/Program Files (x86)/Netapp/Policy/logs/app.log"
    type => "app"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
      charset => "ISO-8859-1"
    }
    start_position => "beginning"
    sincedb_path => "C:/setup/tools/logstash/sincedb_app"
	sincedb_clean_after => 43200
	close_older => 3600
    ignore_older => 0 # Ignore files older than 3 days
  }
  file {
    path => "C:/Program Files (x86)/Netapp/Policy/logs/app.log.*"
mode => "read"
    type => "app_rotate"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
      charset => "ISO-8859-1"
    }
    start_position => "beginning"
    sincedb_path => "C:/setup/tools/logstash/sincedb_app_rotate"
	sincedb_clean_after => 43200
	close_older => 3600
    ignore_older => 259200 # Ignore files older than 3 days
  }
}
}

this will read all the file app.log and app.log.1, app.log.2 .....
file lock will also not happen when reading the archive file as I m using the read mode.

6

This second file input is not needed in most of situations and may lead to duplication.

Logstash can handle file rotation, your issue is that when the file is rotate it seems that the permissions are being changed, so logstash can not read anymore from the previous file, now rotated to .1.

This needs to be fixed outside logstash, if it is not possible you may need to the path to app.log.1 in your file input, but do not change from tail mode to read mode as this can lead to duplication.

7

@leandrojmp I see one observation here when I try to open app.log.1 file. If I try to stop logstash and open the file. I can do it. That's the reason I m saying logstash locks the file.

Please advise.
app.log rename to app.log.1. logstash should still able to tail app.log not sure why this is not happening.

input {
  file {
    path => "C:/Program Files (x86)/Netapp/Policy/logs/app.log"
    type => "app"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
      charset => "ISO-8859-1"
    }
    start_position => "beginning"
    sincedb_path => "C:/setup/tools/logstash/sincedb_app"
	sincedb_clean_after => 43200
	close_older => 3600
  }
}
8

If Logstash is reading app.log and this is rotated to app.log.1, logstash will finishes processing the rotated file, app.log.1 and then start reading app.log.

I'm not sure exactly how this file is being rotated, if the permissions changes during rotation, then logstash will not work as expected.

From the log you shared it seems that the permissions may be changed and logstash cannot finish reading the file after it is rotated.

9

@Thanks for helping. I m confused completely. I m using latest logstash 8.17.3. Windows server

input {
  file {
    path => "C:/Program Files (x86)/Netapp/Policy/logs/app.log"
    type => "app"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
      charset => "ISO-8859-1"
    }
    start_position => "beginning"
    sincedb_path => "C:/setup/tools/logstash/sincedb_app"
	sincedb_clean_after => 43200
	close_older => 3600
  }
}

Before logstash restart

sincedb

3336151636-421785-458752 0 0 10487888 1745520037.272 C:/Program Files (x86)/Netapp/Policy/logs/app.log
3336151636-421785-589824 0 0 10485823 1745976815.327 C:/Program Files (x86)/Netapp/Policy/logs/app.log

after logstash restart

3336151636-421785-458752 0 0 10487888 1745520037.272 C:/Program Files (x86)/Netapp/Policy/logs/app.log
3336151636-421785-589824 0 0 10485823 1745976815.327 C:/Program Files (x86)/Netapp/Policy/logs/app.log
3336151636-421785-655360 0 0 4359 1746457399.167 C:/Program Files (x86)/Netapp/Policy/logs/app.log

why the new inode was not updated in sincedb file. After restart it is determining the inode?
how to handle this scenario ?

10

@leandrojmp Please share your thoughts.