Filebeat for logs that send once every couple days

Elastic Stack Beats
1

Hi,

I'm running into some issues with logs I'm shipping with filebeat to logstash. I'm not quite sure what the issue is, but the file with the logs only refreshes once a day or once every two days with one or two lines that I want to send to Logstash.

Data is flowing in on occasion, but usually a day or so late. So the connection works, but it just doesn't seem to work well if the logs are not flowing constantly -- I have another log file on the same machine with the same filebeat that is constantly being appended to and that one is flowing to logstash just fine. Is there some filebeat setting I need to explicitly set in this situation where the log file isn't constantly being appended to? My filebeat is:

- type: log
  tags: ["test"]
  # Change to true to enable this input configuration.
  enabled: true
  # clean_inactive: 15s
  # ignore_older: 8s
  # scan_frequency: 1s
  # close_timeout: 10s
    - C:\Elastic\logs\test.log

As you can tell it's from a windows machine

2

Don't use ignore_older in this case, as it tells Filebeat to ignore files that hasn't been updated for a while. Same goes for clean_inactive. If you enable it, it removes the entry from the registry after the configured interval, so Filebeat reads the file again from the beginning if it encounters it.

Could you please share the debug logs of Filebeat?

3

I would love to but there's a lot of sensitive data in there, is there any suggestion you have without seeing it? I didn't make any additional file beat config changes, and I can see that the harvester is running, it looks into that file and reports that there's no change in it, until it gets to 2019-11-25T05:46:31.734-0800 INFO log/harvester.go:279 File is inactive: C:\Elastic\logs\test.log. Closing because close_inactive of 5m0s reached.... and no errors in it.

4

You could test if you write the log file Filebeat is able to find it. Look for "Start next scan" in the debug logs. If Filebeat cannot pick it up during the scan, something is not right.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.