I have given below lines in filebeat yml.
filebeat.prospectors:
- input_type: log
paths:
- /var/log/*.log
- \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSService\*.log
- D:\logs\*.log
document_type: log
scan_frequency: 5s
output.logstash:
# The Logstash hosts
hosts: ["10.209.68.107:5044"]
index: "filebeat-%{+yyyy.MM.dd}"
I can see only - D:\logs\*.log logs are captured whereas NAS path location logs are not captured.
Please let us know below NAS path is configured properly or not in filebeat.yml.
- \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSService\*.log
Your inputs helps us a lot.I do not know the answer, but found a note about NAS in the Filebeat FAQ.
Please be aware of the link that @Christian_Dahlqvist mentioned.
Nevertheless Filebeat should still pick up some files. Which Filebeat version are you using? Can you share the log output? Your config file indentation looks quite "off". Is this the way it looks in your config file?
im using filebeat version 5.2.
please find the below logs
2017-04-06T12:25:56+05:30 INFO Metrics logging every 30s
2017-04-06T12:25:56+05:30 INFO Home path: [D:\Beats\filebeat-5.2.0-windows-x86_64] Config path: [D:\Beats\filebeat-5.2.0-windows-x86_64] Data path: [C:\\ProgramData\\filebeat] Logs path: [D:\Beats\filebeat-5.2.0-windows-x86_64\logs]
2017-04-06T12:25:56+05:30 INFO Setup Beat: filebeat; Version: 5.2.0
2017-04-06T12:25:56+05:30 DBG Processors:
2017-04-06T12:25:56+05:30 DBG Initializing output plugins
2017-04-06T12:25:56+05:30 INFO Max Retries set to: 3
2017-04-06T12:25:56+05:30 INFO Activated logstash as output plugin.
2017-04-06T12:25:56+05:30 DBG Create output worker
2017-04-06T12:25:56+05:30 DBG No output is defined to store the topology. The server fields might not be filled.
2017-04-06T12:25:56+05:30 INFO Publisher name: BNGWIDAP106
2017-04-06T12:25:56+05:30 INFO Flush Interval set to: 1s
2017-04-06T12:25:56+05:30 INFO Max Bulk Size set to: 2048
2017-04-06T12:25:56+05:30 DBG create bulk processing worker (interval=1s, bulk size=2048)
2017-04-06T12:25:56+05:30 INFO filebeat start running.
2017-04-06T12:25:56+05:30 DBG Windows is interactive: false
2017-04-06T12:25:56+05:30 INFO Registry file set to: C:\ProgramData\filebeat\registry
2017-04-06T12:25:56+05:30 INFO Loading registrar data from C:\ProgramData\filebeat\registry
2017-04-06T12:25:56+05:30 INFO States Loaded from registrar: 368
2017-04-06T12:25:56+05:30 INFO Loading Prospectors: 1
2017-04-06T12:25:56+05:30 INFO Starting Registrar
2017-04-06T12:25:56+05:30 INFO Start sending events to output
2017-04-06T12:25:56+05:30 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-04-06T12:25:56+05:30 DBG exclude_files: []
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-30.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-03.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-20.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\EDSService.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-02-22.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-06.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-08.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-31.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-04-04.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-02-24.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-01.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-10.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-14.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-15.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-28.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\EDSPlugin.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-02-21.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-02-23.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-02-28.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-24.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-02-27.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-02.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-21.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-03-27.log
2017-04-06T12:25:56+05:30 DBG New state added for D:\logs\PublishPlugin_2017-04-03.log
2017-04-06T12:25:56+05:30 DBG New state added for \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log
2017-04-06T12:25:56+05:30 INFO Prospector with previous states loaded: 30
2017-04-06T12:25:56+05:30 DBG File Configs: [/var/log/*.log \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log D:\logs\*.log]
2017-04-06T12:25:56+05:30 INFO Loading Prospectors completed. Number of prospectors: 1
2017-04-06T12:25:56+05:30 INFO All prospectors are initialised and running with 368 states to persist
2017-04-06T12:25:56+05:30 DBG Starting prospector 0
2017-04-06T12:25:56+05:30 INFO Starting prospector of type: log
2017-04-06T12:25:56+05:30 DBG Start next scan
2017-04-06T12:25:56+05:30 DBG Check file for harvesting: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log
2017-04-06T12:25:56+05:30 DBG Update existing file for harvesting: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log, offset: 8017
2017-04-06T12:25:56+05:30 DBG File didn't change: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log
2017-04-06T12:25:56+05:30 DBG Check file for harvesting: D:\logs\PublishPlugin_2017-02-20.log
2017-04-06T12:25:56+05:30 DBG Update existing file for harvesting: D:\logs\PublishPlugin_2017-02-20.log, offset: 13426
2017-04-06T12:25:56+05:30 DBG Resuming harvesting of file: D:\logs\PublishPlugin_2017-02-20.log, offset: 13426
2017-04-06T13:29:03+05:30 DBG Check file for harvesting: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log
2017-04-06T13:29:03+05:30 DBG Update existing file for harvesting: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log, offset: 8017
2017-04-06T13:29:03+05:30 DBG File didn't change: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log
2017-04-06T13:29:03+05:30 DBG Check file for harvesting: D:\logs\EDSPlugin.log
2017-04-06T13:29:03+05:30 DBG Update existing file for harvesting: D:\logs\EDSPlugin.log, offset: 527080
2017-04-06T13:29:03+05:30 DBG File didn't change: D:\logs\EDSPlugin.log
2017-04-06T13:29:03+05:30 DBG Check file for harvesting: D:\logs\PublishPlugin_2017-02-23.log
2017-04-06T13:29:03+05:30 DBG Update existing file for harvesting: D:\logs\PublishPlugin_2017-02-23.log, offset: 21667
2017-04-06T13:29:03+05:30 DBG Harvester for file is still running: D:\logs\PublishPlugin_2017-02-23.log
2017-04-06T13:29:03+05:30 DBG Check file for harvesting: D:\logs\PublishPlugin_2017-03-03.log
Sorry, i dint get what you menat by below statement
Your config file indentation looks quite "off"
In the above log I see:
2017-04-06T13:29:03+05:30 DBG File didn't change: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log
So fielbeat is aware of the file but assumes it didn't change. Can you check if the reported offset correlates with the size of the file? How is the file updated?
About config file "off". With YAML indentation is very important. Looking at your config file above it would not work the way it is shown.
Hi Ruflin,
Sorry i don't know what is reported offset correlates with the size of the file.?
Please explain and the above file which i mentioned is shared path where we store logs in those specific path to monitor our applications.
Please let us know what we must change in the YML file.
Please be aware that we do NOT recommend to use shared drives: https://www.elastic.co/guide/en/beats/filebeat/current/faq.html#filebeat-network-volumes You should install filebeat on all edge nodes.
My assumption is in the above that the meta data from the network drive is not updated, so from the perspective of filebeat the file did not change.
thank you ruflin, As per recommended Will not use network drive in our environment.
Thanks for your help 
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.