I've recently noticed that
filebeat is shipping not only new lines in my logs, but apparently all the lines, all the time.
E.g. I'm shipping nginx logs that are being created very ordinarily:
access_log /var/log/nginx/site-access.log combined;
My filebeat config says:
filebeat: prospectors: - document_type: log fields: gl2_source_collector: bd4ffc76-23b4-49c1-87df-f****** ignore_older: 0 input_type: log paths: - /var/log/nginx/*.log scan_frequency: 10s tail_files: true output: logstash: hosts: - graylog.*****.com:5044 path: data: /var/cache/graylog/collector-sidecar/filebeat/data logs: /var/log/graylog/collector-sidecar tags: - linux - nginx - phpfpm
And yet, I'm seeing log entries being shipped to my graylog instance that are several days old.
Any clues to what could be causing this? To my knowledge, the log files are not truncated other than a logrotate each week.
root@host2:~# stat /var/log/nginx/site-access.log File: ‘/var/log/nginx/site-access.log’ Size: 6121515 Blocks: 11968 IO Block: 4096 regular file Device: 801h/2049d Inode: 662327 Links: 1 Access: (0640/-rw-r-----) Uid: ( 33/www-data) Gid: ( 4/ adm) Access: 2020-01-23 12:49:46.556082523 +0100 Modify: 2020-01-23 12:49:49.560126373 +0100 Change: 2020-01-23 12:49:49.560126373 +0100 Birth: -