Filebeat stop harvesting

Elastic Stack Beats
1

I've configuring filebeat, logstash, elasticsearch. it works pretty well.
But for some reason filebeat will stop harvesting log file. The file will be generated everyday. I've try this for few times at it always happened at certain time.
for example:
i run it at 12 Oct, until it will harvest the file that generated on 13 Oct and 14 Oct, but for some reason it wont harvesting on 15 Oct. Its pretty weird so i check the logstash but the service still running, then i check the filebeat, it still running but it not harvesting file on 15 Oct.
filebeat.yml:

filebeat.inputs:
    - type: log
      paths:
       - c:/inetpub/logs/LogFiles/W3SVC3/*.log    

output.logstash:
  hosts: ["10.64.2.246:5044"]

run :

.\filebeat.exe -c filebeat.YML -e -d "*"

any respond or suggestions will be appreciated

2

@Charles_Yuliansen - It is difficult to tell what the problem could be without any debug logs from Filebeat. I see that you are already running Filebeat with debug logging enabled on all selectors, that would be great if you could share with us the logs (from the moment where you start Filebeat).

I see that you are using Filebeat on Windows, you could be facing an issue related to the file handlers and the Windows file rotation. You can read more about this here.

Could you also tell us if the logs are stored in network volumes? There are some known issues with network volumes.

Thank you.

1 Like
3

Hi there thank you so much for your responds, but seems i dont get it what should i do next. The log file generated everyday, and for first few day filebeat did well by reading new file each day, but for some reason it stop reading new file (the log only shown that nothing changes from the previous file which is obvious because it generated everyday) after certain day.

4

@Charles_Yuliansen - There are a couple of known issues/limitations in Windows that you should be aware of:

To find-out what's going on, it would be good to run Filebeat with debug logging:

  • filebeat -e -d "*"

Once the issue happens, you can attach the debug logs.

I am also wondering which Filebeat version you are currently using?

1 Like
5

Hi there I'm running filebeat 7.4.2. Is there something that I can use to stop filebeat reading from the beginning of the folder again? I mean file generated everyday in the folder, and if i have the file from 1 January 2019 and the latest file is 6 March 2020 it still checking if 1 January 2019 have any change which is impossible in my case. Thank you so much for your reply and advice.

6

@Charles_Yuliansen - I suggest you look into the following parameters:

  • ignore_older, close_inactive and clean_inactive.

Configuring these parameters should prevent Filebeat from reading the older logs (and will also clean the Filebeat registry file accordingly). Note: if your log files are deleted/renamed, there are some other parameters that you could use (e.g close_renamed, etc.)

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.