Harvester could not be started on new file - access is denied

[byoungman (Bill Youngman)]

Version: 6.2.4

Observations:

• I have filebeat watch application logs that contain sub-directories with more log files in the sub-directories.
• Filebeat is running under an account that has read permissions to the log parent folder and child folders
• Filebeat can successfully open and harvest files in every folder save one and for that one I'm seeing the following error:

Harvester could not be started on new file: '\machinename\directory path\log filename', Err: Error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening '\machinename\directory path\log filename': Error creating file '%!s(*uint16=0xc0463ae8c0)': Access is denied.

Needless to say that since filebeat can't access these files it's not able to get them marked as closed either so it just keeps processing away on them consuming resources. This is currently in one of our lower environments and I'm holding off moving it into production where the log activity is much greater.

Any help / suggestions / ideas is greatly appreciated.

TIA,
Bill

[exekias (Carlos Pérez Aradros)]

Hi @byoungman,

I'm wondering if you could skip that file in the config? You can use exclude_files to add regexps on problematic ones:

https://www.elastic.co/guide/en/beats/filebeat/6.2/configuration-filebeat-options.html#exclude-files

Best regards

[byoungman (Bill Youngman)]

I'm thinking that that would be the best approach since upon further research I found that those files in that one directory were in fact actually ingested into Logstash and are showing up in Kibana. They are old log files that would have been marked as closed anyway.

Thanks for the reply and reinforcing my thoughts.

Bill