Filebeat stops harvesting - file didn't change

Elastic Stack Beats
1

Today I found out about, and tried out filebeat for sending logs to Elasticsearch via Logstash, which I already had set up. It seemed to work fine for a bit, until I noticed that FB stops sending logs after a period of activity.

Here's my setup:

  • Ubuntu 14.04
  • Filebeat 1.2.2
  • Logstash 2.2
  • Elasticsearch 1.5.2

Using tcpdump I determined that the instance running Filebeat had completely stopped sending data to the logstash instances, so I then turned on debug logging in Filebeat and discovered that it thinks the logs it's watching are not being updated, even though they are.

I primarily noticed this with my apache logs which are constantly receiving traffic. However, Filebeat reports:

2016-05-06T22:55:25Z DBG scan path /var/log/apache2/access.log
2016-05-06T22:55:25Z DBG Check file for harvesting: /var/log/apache2/access.log
2016-05-06T22:55:25Z DBG Update existing file for harvesting: /var/log/apache2/access.log
2016-05-06T22:55:25Z DBG Not harvesting, file didn't change: /var/log/apache2/access.log

I notice that apache constantly updates the modification time on access.log as it writes to it, and I assume the same happens with the other logs I'm monitoring. It appears that Filebeat is getting confused by this behaviour.

I did some googling and found that this seems to have been a problem for others, but so far I haven't found a solution. I tried setting 'spool_size: 1' for each prospector, but it did nothing. I also set registry_file: .filebeat for each prospector, this is the output of it:

{"/var/log/apache2/access.log":{"source":"/var/log/apache2/access.log","FileStateOS":{"inode":154945,"device":51713}},"/var/log/apache2/error.log":{"source":"/var/log/apache2/error.log","FileStateOS":{"inode":154946,"device":51713}},"/var/log/php/php.log":{"source":"/var/log/php/php.log","FileStateOS":{"inode":271523,"device":51713}},"/var/log/syslog":{"source":"/var/log/syslog","FileStateOS":{"inode":154666,"device":51713}}}

I've tried watching this file to see if it changes, but it doesn't. If I restart filebeat it works for a short while and then starts showing that "Not harvesting, file didn't change" error.

Does anyone have a solution for this?

Thanks,
Guy

2

Are the logs by coincidence on a mounted volume? Filebeat uses the modification date decide if a file should be continued to read, so it is good that apache keeps updating it. What are your values for ignore_older and close_older?

3

This looks very similar to my issue.

4

I have exactly the same issue. Filbeat correctly send events when starting but then stop the logs because Not harvesting, file didn't change even if the log file has change

This seems to occurs on busy file which are update every seconds. Normal files like syslog seems to be correctly handled by filebeat

filebeat: 1.2.3
logstash: 2.3.4-1

5

@leoleoleo any chance to test if you encounter still the same issue with the most recent nightly build? https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/

6

I meant to update this ages ago, but this actually started working for me a bit after I posted this issue. I have no idea what changed, but since then it's been fine.