Filebeat not steaming the sar logs

Elastic Stack Beats
1

I am getting following error when streaming the system sar log file

2016-03-01T00:24:08Z INFO Read line error: file inactive
2016-03-01T23:53:02Z INFO Harvester started for file: /var/log/sa/sar01
2016-03-02T00:23:07Z INFO Read line error: file inactive

The SAR files get overwritten after 30 days, so SAR01 get overwritten on Feb01, March01
Our config file has

ignore_older: 30m

scan_frequency: 60s
 spool_size: 1024  
idle_timeout: 5s

This is affecting our ability to collect sar logs, please suggest what config changes that we need to do or if the scenario is not supported.

thanks
Vinay

2

Why set ignore_older: 30m?

  1. Is the 'active' SAR file being shipped ok by filebeat?

  2. If so, why ship file again after being rotated?

  3. You want to index historical data only, or will this be a general problem in production after historical data have been indexed?

  4. How does logrotation work? Sounds like files are just renamed. filebeat detects file has been moved (but still same file) and will not attempt to resend the file.

3

Thanks Steffens

  1. The sar files are in /var/log/sa/sar<1-30>

A particular sar file get overwritten once every month.

When I first started the filebeat, the sar files were shipped accurately but on March 1st, sar01 got overwritten and that didn't got shipped.

So this a general problem in production has nothing to do with the historical data.

4

We are also facing same problem

ignore_older: 12h
  document_type: log4j
  scan_frequency: 0s

/usr/bin/filebeat[27676]: log.go:157: Read line error: file inactive

Any workaround for this issue?

5

Can you provide some more details? Two notes:

  • The indentation in your config looks off
  • I highly discourage using scan_frequency: 0s as this will lead to high cpu and disk usage. Any reason you set it so low?