# Log Rotation with two events depending on each other **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 10, 2016, 11:23am UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645 "2016-11-10T11:23:49Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![guenther](https://avatars.discourse-cdn.com/v4/letter/g/e36b37/32.png) [@guenther](https://discuss.elastic.co/u/guenther)\ **Post date:** [November 10, 2016, 11:23am UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645/1 "2016-11-10T11:23:49Z")
Hi everybody, I have got a problem with file rotation and logging messages that depends on each other. Here my case: Usually events are written to the logfile in the correct order: my.log ```auto {id: "1", status: "Running"} {id: "1", status: "Finished"} ``` The events are sent to elasticsearch and the latest event override the previous one of the same ID, what is necessary for me. So my problem is, that 1. First log entry is sent to ES 2. File rotation happens -\> my.log is renamed to my.log.1 3. Second log entry is written to new my.log 4. Filebeat correctly harvesting both files, but in that case a few lines of the my.log are sent to ES before the my.log.1 is harvested to the end. my.log.1 ```auto {id: "1", status: "Running"} ``` my.log ```auto {id: "1", status: "Finished"} ``` Here the config (filebeat 5.0) ```auto filebeat.prospectors: # Each - is a prospector. Most options can be set at the prospector level, so # you can use different prospectors for various configurations. # Below are the prospector specific configurations. - input_type: log paths: - /var/log/smec/*.log* encoding: utf-8 document_type: TaskLogEntry scan_frequency: 1s ``` What would be the correct way to read out the old (renamed) log file before start reading the new one? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 10, 2016, 1:06pm UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645/2 "2016-11-10T13:06:27Z")
filebeat tries to harvest log files concurrently. It has no idea about file-order in the presence of log-rotation. ---
**Author:** ![guenther](https://avatars.discourse-cdn.com/v4/letter/g/e36b37/32.png) [@guenther](https://discuss.elastic.co/u/guenther)\ **Post date:** [November 10, 2016, 1:52pm UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645/3 "2016-11-10T13:52:06Z")
Yes, I'm aware of that. But is there a way I can configure filebeat that way, that my problem could be solved? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 10, 2016, 2:04pm UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645/4 "2016-11-10T14:04:27Z")
in filebeat directly? No. In kibana some order is implicit due to '@timestamp' field. ---
**Author:** ![guenther](https://avatars.discourse-cdn.com/v4/letter/g/e36b37/32.png) [@guenther](https://discuss.elastic.co/u/guenther)\ **Post date:** [November 10, 2016, 4:54pm UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645/5 "2016-11-10T16:54:37Z")
Thats too bad that this is not somehow possible in filebeat. In this case, it is not possible in Kibana, because the two log lines have the same ID, and so the latter one overrides former. In my case, sometimes the logical latter one, is sent to ES before the logical former one (in case of a file rotation). That means I don't have a chance with the current filebeat to solve this problem somehow. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 11, 2016, 1:52pm UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645/6 "2016-11-11T13:52:23Z")
Not sure I get you right, but you basically want to display some current state by overwriting entries in elasticsearch? ---
**Author:** ![guenther](https://avatars.discourse-cdn.com/v4/letter/g/e36b37/32.png) [@guenther](https://discuss.elastic.co/u/guenther)\ **Post date:** [November 11, 2016, 2:26pm UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645/7 "2016-11-11T14:26:38Z")
Correct. I have a workflow, and a Step in the workflow create different events. In my simple case it starts with `RUNNING` and ends with `FINISHED`. Only the (current) latest state should be displayed in ES so we just use the same document-id to override any older Documents in ES. But when this two events (`RUNNING` and `FINISHED`) are written to the log file AND the situation occurs, that a file rotations happens exactly in between those two, there is the chance (and sadly this happens quite often), that the new log file is harvested before the old existing one is again harvested to EOF. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 14, 2016, 1:16pm UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645/8 "2016-11-14T13:16:57Z")
As far as I understand your problem, you could solve this by adding your own timestamp and then sorting based on it? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 14, 2016, 1:50pm UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645/9 "2016-11-14T13:50:21Z")
If you have a timestamp associated with your event in the log, although that is not the case in the example you provided, you could send the data to Logstash as I believe it supports scripted updates that would allow you to check the timestamp and only update it it is newer or no document currently exists for that ID. ---
**Author:** ![guenther](https://avatars.discourse-cdn.com/v4/letter/g/e36b37/32.png) [@guenther](https://discuss.elastic.co/u/guenther)\ **Post date:** [November 14, 2016, 7:22pm UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645/10 "2016-11-14T19:22:00Z")
Hi, in fact I have a timestamp available in each log-line. @ruflin: How would sorting solve the problem? I only wan't to see the latest version of a "log line" in ES so I use the same document-id to override older ones. @Christian_Dahlqvist: sounds very nice. I will try this out and let you know if works for me (I guess it will). ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 15, 2016, 10:44am UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645/11 "2016-11-15T10:44:27Z")
@guenther in case you get it working, I'd love to learn about your solution. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 13, 2016, 10:45am UTC](https://discuss.elastic.co/t/log-rotation-with-two-events-depending-on-each-other/65645/12 "2016-12-13T10:45:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.