Wrong file path when file rotation

[caoping]

I was testing how fiebeat handles file roration case. I found two problems, and I'm not sure if that's caused by my configuration or the filebeat issues.

###Problem A: filebeat didn't read any update in the renamed file
Parts of filebeat configuration, and others keep the default settings.
- paths: - /home/work/test/*.log close_older: 1m
What my operations were:

1. created test.log, and appended something into it.

2. mv test.log test1.log
3. created a new test.log and edited it.
4. appended something to test1.log
5. check filebeat output

I found filebeat didn't read the update in test1.log. I checked the register file, it did add test1.log entry and the offset, but no matter how much I added into that old file, the offset didn't change. Is that a issue or caused by me incorrect operation/configuration?
###Problem B: wrong file path in filebeat output
Changed "close_older" to 5 minutes. Then same operations as listed above. But keep in mind that, test1.log was modified within 5 minutes.
close_older: 5m

In this case, filebeat could read any updates in test1.log. But the problem was the file path in its output. Please look at below screenshot.

What I highlighted was the update in test1.log, however, the "source" said it was test.log. Is that a filebeat issue?
Thanks for any help in advance.

[warkolm (Mark Walkom)]

Please don't post pictures of text, they are difficult to read and some people may not be even able to see them.

[caoping]

Hi @warkolm,

I didn't know that before. Thanks for your reminder. But do you have any ideas on my problem? I couldn't figure out why it happened. What's more, filebeat will remove any entries in the register file? If no, its size will become larger and larger over time. If yes, what's the policy when removing?

I'm about to use it in an enterprise grade log solution, so I need to figure out some details. Thank you in advance.

[ruflin]

Some questions:

• Which version of filebeat did you test?
• How did you update the log file with new lines?
• It can happen, that after a rotation the file path is still the path it was when it started reading the file. The file is tracked based on the file handler and not path. Only after some time the path will get updated. If you want to prevent that, you would have to use close_renamed which is available in 5.0.0-alpha4: https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html#_close_renamed But in general it should not be an issue if the "previous path" is still report I would think as rotated files are normally not updated anymore and filebeat only finishes reading.

In the 5.0.0-alpha4 release features were added to clean up the registry file to make sure it does not constantly grow over time. Have a look here: https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html#clean-options

[caoping]

1. filebeat version is 1.2.3
2. I use "cat" to append new lines
3. When log file is renamed, it's possible that filebeat doesn't finish reading. Right? If that, if enable close_renamed, then data loss occurs, which is not accepted for me. Is there any way to tell filebeat to close renamed file when finishing reading, instead of closing it when file renamed?

Also, I have some questions:

1. If there is no update for log file after the period of close_older, will the file handler be closed? If yes, will register file get updated as well?
2. If there is update to a log file after the period of close_older and filebeat closes the file handler, then will it open that file again and read the update? Please see Problem A: filebeat didn't read any update in the renamed file in my post above.

Registry file cleanup will be pretty useful in real world. Thanks for your information.

[ruflin]

2. Appending lines with cat should work as expected. Can you paste the exact lines you did use for testing?
3. By default filebeat finishes a file also if it is renamed. Also if you use close_renamed, it will be finished in case it it is picked up again later by the prospector after scan_frequency. Obviously if the file is not in the prospector pattern or was removed, reading was not finished.

About closing the file: FIlebeat will close the file after close_older (in 5.0 close_inactive) in any case. Isn't that enough in your case?

For the questions:

1. Yes. The registry file gets updated constantly before events are sent. No relation to close_older.
2. Yes, it will pick up the file again after scan_frequency and will continue the reading.

In general please check the docs for what the config options are doing. Let me know if some of the docs are not clear so we can improve it.

[caoping]

In general, the docs are good, and can answer many questions, but it'll be better if there were some details to tell how it works.

1. parts of my test log. I use multilines config section to merge the line not started with ^\[[[:digit:]]\/[[:digit:]]\/[[:digit:]] to the before line.
   [8/5/2016 20:15:34:456 CST] 111111111 2222222222222 [8/5/2016 20:16:23:100 CST] 3333333333 444444444444
   The multilines configuration works well and as expected, however, the filebeat didn't read updates for a renamed file(renamed to test1.log), which was still in the prospector pattern(test*.log). I had waited for the time duration specified by scan_frequency, but still no reading from the renamed file.
2. Per your explanation and the docs, my understanding is that if the renamed file is not in the prospector pattern, then reading might not be finished, which might cause data loss. Otherwise, filebeat will finish reading and then close the file handler. Right? If yes, that makes sense and I have no further question about it.

Just confirm with you that for current filebeat version 1.2.3, the registry file will not clean up state entry and close_* settings have no affect on registry file, correct?