# Issues with NXLog log rotation and Filebeat
**URL:** https://discuss.elastic.co/t/issues-with-nxlog-log-rotation-and-filebeat/297952
**Category:** Beats
**Tags:** filebeat
**Created:** [February 22, 2022, 8:41pm UTC](https://discuss.elastic.co/t/issues-with-nxlog-log-rotation-and-filebeat/297952 "2022-02-22T20:41:57Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@ganesh\_tarone](https://discuss.elastic.co/u/ganesh_tarone)
#### Post date: [February 22, 2022, 8:41pm UTC](https://discuss.elastic.co/t/issues-with-nxlog-log-rotation-and-filebeat/297952/1 "2022-02-22T20:41:57Z")
I am using NXLog to output some events to a .json file.
Basically, I'm using nxlog filecycle() method (find more [here](https://nxlog.co/documentation/nxlog-user-guide/log-rotation.html)) to rotate log files. But the way nxlog does it is as below, (For weekly retention as an example)
- For 1st week, it writes to say A.json file
- After 1st week ends, A.json is renamed to A.json.1 and new file A.json is created where further loggings happen.
- Similarly by end of 2nd week, A.json is renamed to A.json.1 and A.json.1 is renamed to A.json.2 (So eventually we are always writing to A.json)
Now I'm using filebeat to send these logs,
1. In filebeat if I give regex as /\* for the directory it will pick all files and there would be data duplicity.( For ex A.json.1 would have same data as A.json).
2. If I only give A.json to read, say before NXLog rotates A.json to A.json.1, and if filebeat is yet to read the entire file, we will lose data which bot could not read before nxlog rotated it.
I also tried to use filestream type of filebeat with
`rotation.external.strategy.copytruncate: suffix_regex: \.\d$`
**In logs, I saw above feature to be experimental. So I'm not sure I should use it for a long go.**
But I get below error,
`ERROR failed to remove file 'C:\Program Files (x86)\nxlog\data\Output\nxlog-output.json.1': The process cannot access the file because it is being used by another process. `
I'm doomed and out of options. How do I solve this issue?
---
### Author:  [@TiagoQueiroz](https://discuss.elastic.co/u/TiagoQueiroz)
#### Post date: [February 24, 2022, 4:49pm UTC](https://discuss.elastic.co/t/issues-with-nxlog-log-rotation-and-filebeat/297952/2 "2022-02-24T16:49:53Z")
Hi @ganesh_tarone the log rotation you described should work on Filebeat log and filestream inputs.
When you first started Filebeat, how many log files did you have? Were they already rotated?
Could you try setting up Filebeat to with a pattern that matches the file you want to harvest and all of its rotated files, but make sure that when you start filebeat only the main log file exists.
Another thing you can try is to exclude the rotated files with `exclude_files` option.
Have you looked at our documentation regarding log rotation:
- [exclude the rotated files with `exclude_files` option.](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#rotating-logs)
- [Log rotation results in lost or duplicate events | Filebeat Reference [8.0] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/file-log-rotation.html)
If none of this work, please share your input config so we can further investigate.
---
### Author:  [@ganesh\_tarone](https://discuss.elastic.co/u/ganesh_tarone)
#### Post date: [February 25, 2022, 9:07am UTC](https://discuss.elastic.co/t/issues-with-nxlog-log-rotation-and-filebeat/297952/3 "2022-02-25T09:07:24Z")
Hi Tiago,
First I start NXLog, the moment I start, A.json and A.json.1 is created. A.json.1 remains empty untill first week (assuming weekly retention). In 2nd week, A.json is renamed to A.json.1 and A.json.1 is deleted. And new A.json is created where further loggings happen.
So after starting NXLog when A.json and A.json.1 are generated, within a minute, I start filebeat and I think it reads from both files, but A.json.1 file is empty anyway. Now, when the time of rotation comes, when files are renamed and all rotation happens, I get this error
`ERROR failed to remove file 'C:\Program Files (x86)\nxlog\data\Output\nxlog-output.json.1': The process cannot access the file because it is being used by another process.`.
By your other solution to exclude rotated files (i.e all files like .1,.2,.3 etc) that would mean we are directly giving A.json as input to filebeat. Logging is done in real time in A.json by NXLog and filebeat is reading from it. **Now, when time comes to rotate the file, NXLog would immediately rotate it after 1 week and we will have a blank A.json. But what if filebeat still had say 10 lines left to read from A.json. That would mean data loss of those 10 lines, right?**
---
### Author:  [@TiagoQueiroz](https://discuss.elastic.co/u/TiagoQueiroz)
#### Post date: [March 2, 2022, 5:14pm UTC](https://discuss.elastic.co/t/issues-with-nxlog-log-rotation-and-filebeat/297952/4 "2022-03-02T17:14:03Z")
When a file is renamed Filebeat can still read from it, so it will finish reading from the renamed file and also start reading the new file.
NXLog is the one issuing the error:
```auto
ERROR failed to remove file 'C:\Program Files (x86)\nxlog\data\Output\nxlog-output.json.1': The process cannot access the file because it is being used by another process.
```
right?
Which version of Filebeat are you running?
We recently fixed a [bug](https://github.com/elastic/beats/issues/29113) in the filestream that was locking files on Windows.
This seems very similar to what you're experiencing.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 30, 2022, 7:14pm UTC](https://discuss.elastic.co/t/issues-with-nxlog-log-rotation-and-filebeat/297952/5 "2022-03-30T19:14:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.