Filebeat is not sending logs due to ignore_older

Elastic Stack Beats
1

The filebeat stop sending few hours after log files are rotated.
I found out that ignore_older is the reason of not sending data from some files.
I haven't set config ignore_older.
So I disabled by ignore_older: 0 explicitly.

But some reason some servers fixed the issue but some servers is happening with the same reason.

Here is my filebeat config.

  • type: log
    enabled: true
    paths:
    • /var/logs/*_technical.log
      multiline.pattern: '^####'
      multiline.negate: true
      multiline.match: after
      tail_files: true
      close_rename: true
      close_timeout: 30m
      ignore_older: 0
2

ignore_older defaults to zero actually so you don’t need to set it if want it disabled.
What you’re setting is the same as the default you’d get if you didn’t use that setting at all.
It’s not influencing your outcome.

You say your log files are rotated, what is their filename after the rotation, usually after rotation the filenames get changed and something is added to the end of the filename. What are the filenames of your files before and after rotation?

3

Thanks @martinr_ubi for replying.

Logfile name is backend.log.
After log rotated, the file is changed and zipped to "backend_201900709.0.log.zip".

Sometimes restarting filbeat make filebeat sending log to logstash but not always.

As I mentioned in the question, I saw the message of not harvesting the log file in the filebeat debug log even ignore_older is disabled.

4

Your harvested path would not include log files named backend.log or the rotated zipped version. Filebeat doesn’t harvest zip files anyway even if they in the path.

What is the message in the logs you are refering too exactly, your haven’t shared that filebeat log line here but you believe its related.

5

@martinr_ubi Please ignore filename. I changed filename to general.

This is log message "Ignore file because ignore_older reached: /var/log/log_technical.log"
As I mentioned in question, the sending logs are being resumed after restarting log.

This happening in some servers and even in the same server, it was okay yesterday, but it is happening today some times.

6

if you do not set ignore_older and filebeat logs a line saying file was ignored because of ignore_older I would conclude this is a bug.

What version of filebeat is this? Did you successfully use an older version before, where this issue did not show up?
Can you reproduce with an older version?
You should find how someone else can reproduce this on their own computer if possible and open a github issue with the issue and steps to reproduce. If the issue is triggered by your rotation scheme, you should describe what and with which config this rotation is done. like logrotate or something else.

If you can’t provide a repro case maybe the devs will be able to understand what happen but if they repro, might be hard to find the bug.

7

Thanks @martinr_ubi for your kind explanation.

It is very hard to reproduce in other people's computer.

The current symptom is that filebeat stop sending technical/performance log files to logstash after few hours log rotating. And if I restart filebeat, then the filebeat resume sending log files to logstash from time to time but not always.

Thanks for your answer.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.