Filebeat pulling older logs to logstash

i am using file-beat to pull the logs to log-stash and from there to gray-log, its working fine with .csv files and when i am sending .txt files facing issue

example : if one error logged at 6:00 AM that log get pushed to grey-log and another error logged at 9:00 AM and appended on the same log file its pushing older log also to grey-log is there a way to stop that presently i am using below configuration please suggest if any thing missed

ignore_older: 5m
close_inactive: 5m
close_renamed: true
close_removed: true
close_eof: true
clean_inactive: 10m
clean_removed: true
#exclude_lines: ["^|"]
#exclude_lines: ['^(.?|)$']
#exclude_files: ["_Info.txt$","_Debug.txt$","MemCache_Error.txt$","UpdateChatInQMemcache_Error.txt$"]
#multiline.pattern: '^%{TIMESTAMP_ISO8601}'
multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
multiline.negate: true
multiline.match: after

I think your issue is that you use clean_inactive: 10m so each file which is read more then 10m ago the state will be removed and on update read from scratch again.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.