Filebeat not sending NAS path logs


(Kumar) #1
I have given below lines in filebeat yml.

filebeat.prospectors:
- input_type: log
paths:
    - /var/log/*.log
    - \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSService\*.log
    - D:\logs\*.log

document_type: log
scan_frequency: 5s

output.logstash:
  # The Logstash hosts
  hosts: ["10.209.68.107:5044"]
  index: "filebeat-%{+yyyy.MM.dd}"

I can see only  - D:\logs\*.log logs are captured whereas NAS path location logs are not captured.

Please let us know below NAS path is configured properly or not in filebeat.yml.

- \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSService\*.log

Your inputs helps us a lot.

(Christian Dahlqvist) #2

I do not know the answer, but found a note about NAS in the Filebeat FAQ.


(ruflin) #3

Please be aware of the link that @Christian_Dahlqvist mentioned.

Nevertheless Filebeat should still pick up some files. Which Filebeat version are you using? Can you share the log output? Your config file indentation looks quite "off". Is this the way it looks in your config file?


(Kumar) #4

im using filebeat version 5.2.

please find the below logs

2017-04-06T12:25:56+05:30 INFO Metrics logging every 30s
2017-04-06T12:25:56+05:30 INFO Home path: [D:\Beats\filebeat-5.2.0-windows-x86_64] Config path: [D:\Beats\filebeat-5.2.0-windows-x86_64] Data path: [C:\\ProgramData\\filebeat] Logs path: [D:\Beats\filebeat-5.2.0-windows-x86_64\logs]
2017-04-06T12:25:56+05:30 INFO Setup Beat: filebeat; Version: 5.2.0
2017-04-06T12:25:56+05:30 DBG  Processors: 
2017-04-06T12:25:56+05:30 DBG  Initializing output plugins
2017-04-06T12:25:56+05:30 INFO Max Retries set to: 3
2017-04-06T12:25:56+05:30 INFO Activated logstash as output plugin.
2017-04-06T12:25:56+05:30 DBG  Create output worker
2017-04-06T12:25:56+05:30 DBG  No output is defined to store the topology. The server fields might not be filled.
2017-04-06T12:25:56+05:30 INFO Publisher name: BNGWIDAP106
2017-04-06T12:25:56+05:30 INFO Flush Interval set to: 1s
2017-04-06T12:25:56+05:30 INFO Max Bulk Size set to: 2048
2017-04-06T12:25:56+05:30 DBG  create bulk processing worker (interval=1s, bulk size=2048)
2017-04-06T12:25:56+05:30 INFO filebeat start running.
2017-04-06T12:25:56+05:30 DBG  Windows is interactive: false
2017-04-06T12:25:56+05:30 INFO Registry file set to: C:\ProgramData\filebeat\registry
2017-04-06T12:25:56+05:30 INFO Loading registrar data from C:\ProgramData\filebeat\registry
2017-04-06T12:25:56+05:30 INFO States Loaded from registrar: 368
2017-04-06T12:25:56+05:30 INFO Loading Prospectors: 1
2017-04-06T12:25:56+05:30 INFO Starting Registrar
2017-04-06T12:25:56+05:30 INFO Start sending events to output
2017-04-06T12:25:56+05:30 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-04-06T12:25:56+05:30 DBG  exclude_files: []
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-30.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-03.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-20.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\EDSService.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-02-22.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-06.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-08.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-31.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-04-04.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-02-24.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-01.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-10.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-14.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-15.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-28.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\EDSPlugin.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-02-21.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-02-23.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-02-28.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-24.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-02-27.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-02.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-21.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-03-27.log
2017-04-06T12:25:56+05:30 DBG  New state added for D:\logs\PublishPlugin_2017-04-03.log
2017-04-06T12:25:56+05:30 DBG  New state added for \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log
2017-04-06T12:25:56+05:30 INFO Prospector with previous states loaded: 30
2017-04-06T12:25:56+05:30 DBG  File Configs: [/var/log/*.log \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log D:\logs\*.log]
2017-04-06T12:25:56+05:30 INFO Loading Prospectors completed. Number of prospectors: 1
2017-04-06T12:25:56+05:30 INFO All prospectors are initialised and running with 368 states to persist
2017-04-06T12:25:56+05:30 DBG  Starting prospector 0
2017-04-06T12:25:56+05:30 INFO Starting prospector of type: log
2017-04-06T12:25:56+05:30 DBG  Start next scan
2017-04-06T12:25:56+05:30 DBG  Check file for harvesting: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log
2017-04-06T12:25:56+05:30 DBG  Update existing file for harvesting: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log, offset: 8017
2017-04-06T12:25:56+05:30 DBG  File didn't change: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log
2017-04-06T12:25:56+05:30 DBG  Check file for harvesting: D:\logs\PublishPlugin_2017-02-20.log
2017-04-06T12:25:56+05:30 DBG  Update existing file for harvesting: D:\logs\PublishPlugin_2017-02-20.log, offset: 13426
2017-04-06T12:25:56+05:30 DBG  Resuming harvesting of file: D:\logs\PublishPlugin_2017-02-20.log, offset: 13426
2017-04-06T13:29:03+05:30 DBG  Check file for harvesting: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log
2017-04-06T13:29:03+05:30 DBG  Update existing file for harvesting: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log, offset: 8017
2017-04-06T13:29:03+05:30 DBG  File didn't change: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log
2017-04-06T13:29:03+05:30 DBG  Check file for harvesting: D:\logs\EDSPlugin.log
2017-04-06T13:29:03+05:30 DBG  Update existing file for harvesting: D:\logs\EDSPlugin.log, offset: 527080
2017-04-06T13:29:03+05:30 DBG  File didn't change: D:\logs\EDSPlugin.log
2017-04-06T13:29:03+05:30 DBG  Check file for harvesting: D:\logs\PublishPlugin_2017-02-23.log
2017-04-06T13:29:03+05:30 DBG  Update existing file for harvesting: D:\logs\PublishPlugin_2017-02-23.log, offset: 21667
2017-04-06T13:29:03+05:30 DBG  Harvester for file is still running: D:\logs\PublishPlugin_2017-02-23.log
2017-04-06T13:29:03+05:30 DBG  Check file for harvesting: D:\logs\PublishPlugin_2017-03-03.log

Sorry, i dint get what you menat by below statement
Your config file indentation looks quite "off"


(ruflin) #5

In the above log I see:

2017-04-06T13:29:03+05:30 DBG  File didn't change: \\BNGWIDFL001.aonnet.aon.net\ECMLogs\NavigatorLogs\EDSPlugin\AonECMEDSPlugin.log

So fielbeat is aware of the file but assumes it didn't change. Can you check if the reported offset correlates with the size of the file? How is the file updated?

About config file "off". With YAML indentation is very important. Looking at your config file above it would not work the way it is shown.


(Kumar) #6

Hi Ruflin,

Sorry i don't know what is reported offset correlates with the size of the file.?

Please explain and the above file which i mentioned is shared path where we store logs in those specific path to monitor our applications.

Please let us know what we must change in the YML file.


(ruflin) #7

Please be aware that we do NOT recommend to use shared drives: https://www.elastic.co/guide/en/beats/filebeat/current/faq.html#filebeat-network-volumes You should install filebeat on all edge nodes.

My assumption is in the above that the meta data from the network drive is not updated, so from the perspective of filebeat the file did not change.


(Kumar) #8

thank you ruflin, As per recommended Will not use network drive in our environment.

Thanks for your help :slight_smile:


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.