Filebeat not harvesting files from subdirectories


(Bill Youngman) #1

Good Afternoon All,

I have filebeat set up to harvest files in the log directory of an application and the application developers just changed the logging format to include sub-directories containing client specific information.

I modified my prospector paths to be \machine-name\drive$\company\application\log** and I did this for each machine in the application cluster.

Under the log directory than can be any number of sub-directories for example:

  • admin
  • Partition7000007
  • Partition12345

New sub-directories are created every time a new partition is created for a client and within each sub-directory client specific logging information is contained.

The problem I'm running into is that all of the log files in the log root directory are being harvested just fine but none of the files in any of the sub-directories are being harvested.

Setting filebeat logging level to 'debug' I can see that the prospector is seeing all of the sub-directories but it is skipping them even when there are log files present so my question is what am I doing wrong? Do I have my prospector paths declared wrong?

TIA,
Bill Youngman


(Christian Stockhaus) #2

which version of the filebeat you are using?

it is not possible in version 5.x.
the option came with the 6.x version of filebeat and is default enabled

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#recursive_glob


(Bill Youngman) #3

I am using 5.6.3.

Is the latest version of Filebeat (6.2.2) backwards compatible with Logstash 5.6.3? As an enterprise we can't fully migrate to 6.2.2 because we have some user facing pages that are dependent on the 5.6.3 templates that were created that wouldn't be compatible with the changes made to v6.xx. The work to migrate them hasn't been scheduled yet.

Thanks


(Bill Youngman) #4

Stood up a test environment and set Filebeat to harvest logs on my local machine and it successfully traversed the root and all sub-directories harvesting the log files contained in them.

Now to start applying this to my network shares - that will be the true test.

Thanks,
Bill


(ruflin) #5

Glad you got it working. Please be aware that it's not recommend to read files from Network shares but directly on the machine where they are produced.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.