I have filebeat set up to harvest files in the log directory of an application and the application developers just changed the logging format to include sub-directories containing client specific information.
I modified my prospector paths to be \machine-name\drive$\company\application\log** and I did this for each machine in the application cluster.
Under the log directory than can be any number of sub-directories for example:
admin
Partition7000007
Partition12345
New sub-directories are created every time a new partition is created for a client and within each sub-directory client specific logging information is contained.
The problem I'm running into is that all of the log files in the log root directory are being harvested just fine but none of the files in any of the sub-directories are being harvested.
Setting filebeat logging level to 'debug' I can see that the prospector is seeing all of the sub-directories but it is skipping them even when there are log files present so my question is what am I doing wrong? Do I have my prospector paths declared wrong?
Is the latest version of Filebeat (6.2.2) backwards compatible with Logstash 5.6.3? As an enterprise we can't fully migrate to 6.2.2 because we have some user facing pages that are dependent on the 5.6.3 templates that were created that wouldn't be compatible with the changes made to v6.xx. The work to migrate them hasn't been scheduled yet.
Stood up a test environment and set Filebeat to harvest logs on my local machine and it successfully traversed the root and all sub-directories harvesting the log files contained in them.
Now to start applying this to my network shares - that will be the true test.
Glad you got it working. Please be aware that it's not recommend to read files from Network shares but directly on the machine where they are produced.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.