Filebeat not working on a particular path

[ashu_29516]

Hi Everyone,

I'm new to Elasticsearch and have been facing an issue with filebeats where I've been observing that logs from certain paths do not seem to get picked when the file in that path is updated. Running filebeat in debugger mode also doesn't show up anything relevant to that path.

However, upon deleting the registry file, filebeat detects the file. This I can say as I see a log message that the harvester has started on the file.

A snippet of how the YAML configuration for the concerned path appear is below.

    - type: log
      enabled: true
      paths:
         - /data/hddata*/hadoop-yarn/container/application_*/container*/*
      fields:
         log_type: log
      ignore_older: 2h
      scan_frequency: 1s
      close_inactive: 1m
      close_timeout: 5m
      fields_under_root: true

[legoguy1000 (Alex)]

I would check the settings for how u have the close_* options or the ignore_older. Log input | Filebeat Reference [7.12] | Elastic.

[ashu_29516]

Hi Alex,

Thanks for sharing this.

I'm not sure if ignore_older is the issue as I'm manually changing the file in the concerned path while monitoring real time filebeat logs. Still I don't see the harvester detecting the changes.

Not sure, but is it possible that the state for the file is causing an issue here and if so using clean_inactive can solve it? However, would using clean_inactive also mean that this would cause the entire file to be ingested again including the lines which were already ingested before?

[legoguy1000 (Alex)]

that i don't know, i would have to refer you to the docs on that.