Issues with NXLog log rotation and Filebeat

I am using NXLog to output some events to a .json file.

Basically, I'm using nxlog filecycle() method (find more here) to rotate log files. But the way nxlog does it is as below, (For weekly retention as an example)

  • For 1st week, it writes to say A.json file
  • After 1st week ends, A.json is renamed to A.json.1 and new file A.json is created where further loggings happen.
  • Similarly by end of 2nd week, A.json is renamed to A.json.1 and A.json.1 is renamed to A.json.2 (So eventually we are always writing to A.json)

Now I'm using filebeat to send these logs,

  1. In filebeat if I give regex as /* for the directory it will pick all files and there would be data duplicity.( For ex A.json.1 would have same data as A.json).

  2. If I only give A.json to read, say before NXLog rotates A.json to A.json.1, and if filebeat is yet to read the entire file, we will lose data which bot could not read before nxlog rotated it.

I also tried to use filestream type of filebeat with
rotation.external.strategy.copytruncate: suffix_regex: \.\d$
In logs, I saw above feature to be experimental. So I'm not sure I should use it for a long go.

But I get below error,
ERROR failed to remove file 'C:\Program Files (x86)\nxlog\data\Output\nxlog-output.json.1': The process cannot access the file because it is being used by another process.

I'm doomed and out of options. How do I solve this issue?

Hi @ganesh_tarone the log rotation you described should work on Filebeat log and filestream inputs.

When you first started Filebeat, how many log files did you have? Were they already rotated?

Could you try setting up Filebeat to with a pattern that matches the file you want to harvest and all of its rotated files, but make sure that when you start filebeat only the main log file exists.

Another thing you can try is to exclude the rotated files with exclude_files option.

Have you looked at our documentation regarding log rotation:

If none of this work, please share your input config so we can further investigate.

Hi Tiago,

First I start NXLog, the moment I start, A.json and A.json.1 is created. A.json.1 remains empty untill first week (assuming weekly retention). In 2nd week, A.json is renamed to A.json.1 and A.json.1 is deleted. And new A.json is created where further loggings happen.

So after starting NXLog when A.json and A.json.1 are generated, within a minute, I start filebeat and I think it reads from both files, but A.json.1 file is empty anyway. Now, when the time of rotation comes, when files are renamed and all rotation happens, I get this error

ERROR failed to remove file 'C:\Program Files (x86)\nxlog\data\Output\nxlog-output.json.1': The process cannot access the file because it is being used by another process..

By your other solution to exclude rotated files (i.e all files like .1,.2,.3 etc) that would mean we are directly giving A.json as input to filebeat. Logging is done in real time in A.json by NXLog and filebeat is reading from it. Now, when time comes to rotate the file, NXLog would immediately rotate it after 1 week and we will have a blank A.json. But what if filebeat still had say 10 lines left to read from A.json. That would mean data loss of those 10 lines, right?

When a file is renamed Filebeat can still read from it, so it will finish reading from the renamed file and also start reading the new file.

NXLog is the one issuing the error:

ERROR failed to remove file 'C:\Program Files (x86)\nxlog\data\Output\nxlog-output.json.1': The process cannot access the file because it is being used by another process.

right?

Which version of Filebeat are you running?

We recently fixed a bug in the filestream that was locking files on Windows.

This seems very similar to what you're experiencing.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.