Filebeat read log4j2 rotated log file again


#1

Hi, I am using filebeat 6.1.3 in windows to read log files from log4j2 to logstash.
When the logfile ratated from xxx.log -> xxx.log.1
logstash received the lines in file xxx.log.1 again
so all the log is read twice.

Can you please tell me if anything wrong in my configs?


log4j2.xml:
<RollingFile name="file2"
append="true"
fileName="C:/xxx/log/log4j2-test.log"
filePattern="C:/xxx/log/log4j2-test.log.%i">
<PatternLayout pattern="${ptn}" />
<Policies>
<SizeBasedTriggeringPolicy size="10 KB" />
</Policies&lgt;
<DefaultRolloverStrategy fileIndex="min" min="1" max="5" />
</RollingFile>


filebeat.yml:

  • type: log
    paths:

    • C:/xxx/log/log4j2-test.log*

    close_renamed: true
    close_removed: true

output.logstash:
hosts: ["localhost:5044"]


logstash conf:

beats {
port => 5044
}


(Carlos PĂ©rez Aradros) #2

Hi @Denshou,

By adding a * at the end of C:/xxx/log/log4j2-test.log you are instructing Filebeat to read .1 file too, so it takes it as a new file and sends everything. Perhaps you wan to drop the *?

Best regards


#3

Hi, exekias, thanks very much!
I tested the setting but found that when the log file rotated
in the interval of the file beat, the rotated logs is lost.
eg:
log1
log2
log3 -> file beat output to logstash here with 1,2,3
log4
log5 -> here, .log ratated to .log.1
the following log is output to new .log file
log6
log7 -> file beat output to logstash here with 6,7
and the log 4,5 is missed


(system) #4

This topic was automatically closed after 21 days. New replies are no longer allowed.