Hello. I use filebeat->logstash->elasticsearch <- kibana . I am facing an issue with filebeat, it doesn't harvest logfile, even at first run. I tried to stop filebeat, remove registry file and start filebeat again without success. I am getting this message in filebeat log with debug level:
2019-02-07T15:57:32.947+0200 DEBUG [input] log/input.go:404 Check file for harvesting: /pathtolog/log.txt 2019-02-07T15:57:32.947+0200 DEBUG [input] log/input.go:555 Ignore file because ignore_older reached: /pathtolog/log.txt 2019-02-07T15:57:32.947+0200 DEBUG [input] file/states.go:68 New state added for /pathtolog/log.txt
So filebeat thinks that ignore_older setting is set, but it is not. I have tried to set it to 0 and 2160h. It didn't help. Also I have tried to use close_eof: true setting, no succes. Every time I removed registry file. Log file has been modified 2019-01-23 last time. Os version: Red Hat Enterprise Linux Server release 7.6 (Maipo). What else can I try?
my filebeat.yml:
output.logstash: enabled: true hosts: ["hostname:5044"] loadbalance: false worker: 10 compression_level: 9 escape_html: true ssl.certificate_authorities: ["/etc/ca.crt"] ssl.certificate: "/etc/beat.crt" ssl.key: "/etc/beat.key" bulk_max_size: 2048 output.elasticsearch: enabled: false hosts: ["hostname:9200"] filebeat.config.modules: enabled: true reload.enabled: true reload.period: 30s path: ${path.config}/modules.d/*.yml queue.mem: events: 4096 flush.min_events: 512 flush.timeout: 5s logging.metrics.enabled: false logging.json: false #logging.level: info logging.level: debug logging.to_files: true logging.files: path: /var/log/filebeat name: filebeat keepfiles: 7 permissions: 0644 rotateeverybytes: 10485760 # = 10MB filebeat.inputs: - type: log enabled: true # close_eof: true # ignore_older: 0 ignore_older: 2160h tail_files: true paths: - /pathtolog/log.txt exclude_files: ['\.gz$','\.zip$','\.tmp$','\.hprof$'] symlinks: true multiline.pattern: '(^\w{3}[[:space:]][0-9]{1,},[[:space:]][0-9]{4}[[:space:]][0-9]{1,}:[0-9]{1,}:[0-9]{1,})|(^\[\w{3}[[:space:]]\w{3}[[:space:]][0-9]{1,}[[:space:]][0-9]{1,}:[0-9]{1,}:[0-9]{1,})|(^\[[0-9]{1,}\/\w{3}\/[0-9]{1,}:[0-9]{1,}:[0-9]{1,})|(^[0-9]{4}-[0-9]{1,2 }-[0-9]{1,2}[[:space:]][0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2})|(^[0-9]{4}-[0-9]{1,2}-[0-9]{1,2}[[:space:]][0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}.[0-9]{1,3})|([0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2},[0-9]{3})|(^[0-9]{1,2}.[0-9]{1,2}.[0-9]{4}[[:space:]][0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2})' multiline.negate: false multiline.match: before