# Processing log files while using logrotate
**URL:** https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174
**Category:** Logstash
**Created:** [January 26, 2016, 11:03pm UTC](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174 "2016-01-26T23:03:35Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@rspangle1](https://discuss.elastic.co/u/rspangle1)
#### Post date: [January 26, 2016, 11:03pm UTC](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174/1 "2016-01-26T23:03:36Z")
I was wondering what everybody's opinion is on processing log files read by logstash but need to be rotated using logrotate. It appears that the two don't work very well together. I'm trying to make sure we don't loose any log data during the log rotation but it appears that if we stop ELK we can't make sure it reads to the end of the file. I did see in the documentation where it says to add multiple log files in the Input - File section, but it seems like there still might be a problem. Thanks
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 27, 2016, 8:27am UTC](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174/2 "2016-01-27T08:27:17Z")
Logstash's behavior when logs are rotated is very much underdocumented. There are probably cases where rotations aren't dealt with gracefully.
> It appears that the two don't work very well together.
Could you be more specific?
---
### Author:  [@rspangle1](https://discuss.elastic.co/u/rspangle1)
#### Post date: [January 27, 2016, 12:47pm UTC](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174/3 "2016-01-27T12:47:22Z")
If I understand logrotate correctly as it relates to logstash specifically, here is my example: if you are rotating a file called snare.log, logrotate will either make a copy of the file and then truncate the original file or just make a copy of the file and leave the original alone. So, after the first iteration of logrotate, you would have snare.log and snare.log.1 and snare.log may be empty or not depending on what parameter you used. If you stopped ELK before logrotate started and then restarted it after logrotate finished, you need to finish processing anything in the original snare.log BEFORE you start processing what is in snare.log.1. If I'm reading the documentation of Logstash -\> Input -\> File correctly, there is a chance that you might miss some log entries. I'm including the link here:
[https://www.elastic.co/guide/en/logstash/1.5/plugins-inputs-file.html#start\_position](https://www.elastic.co/guide/en/logstash/1.5/plugins-inputs-file.html#start_position)
Thanks
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 27, 2016, 1:14pm UTC](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174/4 "2016-01-27T13:14:30Z")
> If I understand logrotate correctly as it relates to logstash specifically, here is my example: if you are rotating a file called snare.log, logrotate will either make a copy of the file and then truncate the original file or just make a copy of the file and leave the original alone.
Copy & truncate is indeed one way of rotating, but I've never heard of rotation by merely copying. What would be the point of that? The other dominant way of rotating is to rename the log file and either let the program discover the rotation by itself so that it can reopen the file, or HUPing or otherwise restarting or reloading the program.
> If you stopped ELK before logrotate started and then restarted it after logrotate finished, you need to finish processing anything in the original snare.log BEFORE you start processing what is in snare.log.1.
You mean that Logstash should read the remainder of (what's now) snare.log.1 before starting with the (more or less empty) snare.log? Yes, if you have order-dependent filtering of events.
---
### Author:  [@rspangle1](https://discuss.elastic.co/u/rspangle1)
#### Post date: [January 27, 2016, 1:17pm UTC](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174/5 "2016-01-27T13:17:37Z")
correct, can you explain what you mean by order-dependent filtering of events?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [January 27, 2016, 1:19pm UTC](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174/6 "2016-01-27T13:19:03Z")
Sorry. I meant filters like multiline, aggregate, or similar that care about the processing order. For independent events it doesn't matter in which order events are processed.
---
### Author:  [@rspangle1](https://discuss.elastic.co/u/rspangle1)
#### Post date: [January 27, 2016, 1:28pm UTC](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174/7 "2016-01-27T13:28:20Z")
I think part of the problem is the since\_db file. Using my earlier example, if I have snare.log:
1. rsyslog is writing to snare.log
2. At the same time logstash is reading thru it and updating the since\_db file for snare.log
3. Stop ELK and then run logrotate (which forces rsyslog to stop momentarily)
4. logrotate copies what's in snare.log to snare.log.1 and truncates snare.log and then restarts rsyslog
5. When ELK restarts, does it see snare.log.1 as the file it was reading previously as snare.log and does it use the same since\_db?
I know the since\_db uses the inode and byte offset to know where it is in the file (assuming the inode number follows the new file name but I think rsyslog uses the file handle to keep track of files.
Thanks for the dialog on this!!!
---
### Author:  [@Maya\_SH](https://discuss.elastic.co/u/Maya_SH)
#### Post date: [June 13, 2016, 8:05pm UTC](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174/8 "2016-06-13T20:05:48Z")
Hi
I am really confused. I have read probably every post about this but came out with different conclusion every time !
consider (path =\> "mylog.log\*") && start\_position =\> "end". Now if I have a log file called "mylog.log" with size Y
at some point in time logstash is reading at position X of "mylog.log" and logrotate executed the following (1.) cp mylog.log mylog.log1 (2.) echo "" \> mylog.log
question is: what will happen to bytes X--\>Y will logstash read them or not ??
---
### Author:  [@memelet](https://discuss.elastic.co/u/memelet)
#### Post date: [July 26, 2016, 8:52am UTC](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174/9 "2016-07-26T08:52:33Z")
I too would like to understand how not to lose data in the face logrotation.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:46am UTC](https://discuss.elastic.co/t/processing-log-files-while-using-logrotate/40174/10 "2017-07-06T04:46:29Z")