Filebeat not picking logs when run as service

[pavanssetty]

I am using Filebeat 5.4 version on Windows Server 2012.
When I run filebeat from PowerShell or Command Prompt it is shipping logs successfully to my logstash instance running on Command Prompt.
But when I try installing it as a service, it is not shipping any logs to logstash as if it is not reading my filebeat.yml file.
There is no message in logs that it is not able to locate the filebeat.yml file

Here is the Path to execute from the service:
C:\Program Files\Filebeat\filebeat.exe" -c "C:\Program Files\Filebeat\filebeat.yml" -path.home "C:\Program Files\Filebeat" -path.data "C:\Program Files\Filebeat\data"

Below is the log when run as service:

2017-07-18T19:04:43+02:00 INFO Home path: [C:\Program Files\Filebeat] Config path: [C:\Program Files\Filebeat] Data path: [C:\Program Files\Filebeat\data] Logs path: [C:\Program Files\Filebeat\logs]
2017-07-18T19:04:43+02:00 INFO Setup Beat: filebeat; Version: 5.4.0
2017-07-18T19:04:43+02:00 INFO Max Retries set to: 3
2017-07-18T19:04:43+02:00 INFO Activated logstash as output plugin.
2017-07-18T19:04:43+02:00 INFO Publisher name: DEHEREMAP9769
2017-07-18T19:04:43+02:00 INFO Flush Interval set to: 1s
2017-07-18T19:04:43+02:00 INFO Max Bulk Size set to: 2048
2017-07-18T19:04:43+02:00 INFO filebeat start running.
2017-07-18T19:04:43+02:00 INFO Registry file set to: C:\Program Files\Filebeat\data\registry
2017-07-18T19:04:43+02:00 INFO Loading registrar data from C:\Program Files\Filebeat\data\registry
2017-07-18T19:04:43+02:00 INFO States Loaded from registrar: 3
2017-07-18T19:04:43+02:00 INFO Loading Prospectors: 1
2017-07-18T19:04:43+02:00 INFO Prospector with previous states loaded: 0
2017-07-18T19:04:43+02:00 INFO Starting prospector of type: log; id: 17893974800932756507
2017-07-18T19:04:43+02:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-07-18T19:04:43+02:00 INFO Metrics logging every 30s
2017-07-18T19:04:43+02:00 INFO Starting Registrar
2017-07-18T19:04:43+02:00 INFO Start sending events to output
2017-07-18T19:04:43+02:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s

I am not able to figure out what is missing here.

[andrewkroh]

You could enable debug logging to see if this provides any more clues. Also please share your config file.

> Stop-Service filebeat
# Add "logging.level: debug" to the config file.
> Start-Service filebeat

[pavanssetty]

Here is the log now. IT looks like it is able to read the filebeat.yml file correctly, but not able to read the log files at the network path

2017-07-19T13:52:37+02:00 DBG Disable stderr logging
2017-07-19T13:52:37+02:00 INFO Home path: [C:\Program Files\Filebeat] Config path: [C:\Program Files\Filebeat] Data path: [C:\Program Files\Filebeat\data] Logs path: [C:\Program Files\Filebeat\logs]
2017-07-19T13:52:37+02:00 INFO Setup Beat: filebeat; Version: 5.4.0
2017-07-19T13:52:37+02:00 DBG Processors:
2017-07-19T13:52:37+02:00 DBG Initializing output plugins
2017-07-19T13:52:37+02:00 INFO Max Retries set to: 3
2017-07-19T13:52:37+02:00 INFO Activated logstash as output plugin.
2017-07-19T13:52:37+02:00 DBG Create output worker
2017-07-19T13:52:37+02:00 DBG No output is defined to store the topology. The server fields might not be filled.
2017-07-19T13:52:37+02:00 INFO Publisher name: DEHEREMAP9769
2017-07-19T13:52:37+02:00 INFO Flush Interval set to: 1s
2017-07-19T13:52:37+02:00 INFO Max Bulk Size set to: 2048
2017-07-19T13:52:37+02:00 DBG create bulk processing worker (interval=1s, bulk size=2048)
2017-07-19T13:52:37+02:00 INFO filebeat start running.
2017-07-19T13:52:37+02:00 INFO Registry file set to: C:\Program Files\Filebeat\data\registry
2017-07-19T13:52:37+02:00 INFO Loading registrar data from C:\Program Files\Filebeat\data\registry
2017-07-19T13:52:37+02:00 INFO States Loaded from registrar: 3
2017-07-19T13:52:37+02:00 INFO Loading Prospectors: 1
2017-07-19T13:52:37+02:00 DBG File Configs: [Z:\.nasuni\audit\<<InternalFolderName>>\\.csv]
2017-07-19T13:52:37+02:00 DBG exclude_files: []
2017-07-19T13:52:37+02:00 INFO Prospector with previous states loaded: 0
2017-07-19T13:52:37+02:00 INFO Starting prospector of type: log; id: 17893974800932756507
2017-07-19T13:52:37+02:00 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-07-19T13:52:37+02:00 INFO Metrics logging every 30s
2017-07-19T13:52:37+02:00 DBG Windows is interactive: false
2017-07-19T13:52:37+02:00 INFO Starting Registrar
2017-07-19T13:52:37+02:00 INFO Start sending events to output
2017-07-19T13:52:37+02:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-07-19T13:52:37+02:00 DBG Start next scan

My config file: filebeat.yml

filebeat.prospectors:

- input_type: log

  paths:
  
    - Z:\.nasuni\audit\<<InternalFolderName>>\*\*.csv
    
  scan_frequency: 60s

  exclude_lines: [".*Internal.*"]

output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

[andrewkroh]

Are there any entries in the registry file (C:\Program Files\Filebeat\data\registry) for the csv files you are trying to monitor with Filebeat?

In general we do not recommend using Filebeat with network mounts. https://www.elastic.co/guide/en/beats/filebeat/current/faq.html#filebeat-network-volumes

[ogauchard]

Sounds like the account used to run the windows service is not allowed to read target directory on network mount.

[pavanssetty]

Yes I was trying to read the directory using windows mount. Thank You for pointing it.