Filebeat is shipping old log lines

[kwisatz]

I've recently noticed that filebeat is shipping not only new lines in my logs, but apparently all the lines, all the time.

E.g. I'm shipping nginx logs that are being created very ordinarily:

  access_log            /var/log/nginx/site-access.log combined;

My filebeat config says:

filebeat:
  prospectors:
  - document_type: log
    fields:
      gl2_source_collector: bd4ffc76-23b4-49c1-87df-f******
    ignore_older: 0
    input_type: log
    paths:
    - /var/log/nginx/*.log
    scan_frequency: 10s
    tail_files: true
output:
  logstash:
    hosts:
    - graylog.*****.com:5044
path:
  data: /var/cache/graylog/collector-sidecar/filebeat/data
  logs: /var/log/graylog/collector-sidecar
tags:
- linux
- nginx
- phpfpm

And yet, I'm seeing log entries being shipped to my graylog instance that are several days old.
Any clues to what could be causing this? To my knowledge, the log files are not truncated other than a logrotate each week.

E.g.

root@host2:~# stat /var/log/nginx/site-access.log
  File: ‘/var/log/nginx/site-access.log’
  Size: 6121515         Blocks: 11968      IO Block: 4096   regular file
Device: 801h/2049d      Inode: 662327      Links: 1
Access: (0640/-rw-r-----)  Uid: (   33/www-data)   Gid: (    4/     adm)
Access: 2020-01-23 12:49:46.556082523 +0100
Modify: 2020-01-23 12:49:49.560126373 +0100
Change: 2020-01-23 12:49:49.560126373 +0100
 Birth: -

[jsoriano (Jaime Soriano)]

Hey @kwisatz,

The problem might be on the rotation of files, could you check the log.source.path of the old entries shipped? What is your logrotate configuration for these files?

You can check this guide with information about common issues with rotated files: https://www.elastic.co/guide/en/beats/filebeat/7.5/file-log-rotation.html

Oh, and what version of filebeat are you using?

[kwisatz]

Thanks for your reply @jsoriano

In the end, I think the reason we were seeing old log entries was not related to filebeats at all, but rather to our graylog instance (yeah, I know) processing its disk journal after it got stuck on an index for a while.

So I guess this can be marked as closed/invalid.