Shipping logs from multiple files

Yaml is sensitive, make sure your indentations are correct.

Really? How can I make sure? Is there any guideline?

Can you post your entire config somewhere - gist/pastebin/etc?

Here you go:

Pretty sure your issue is here - https://gist.github.com/najamss/8382b64cf403d0eb00901162690469ac#file-filebeat-yml-L205-L210

I am following this document but still issue is there... It's frustrating...

http://ess.khhq.net/wiki/YAML_Tutorial

You need to uncomment line 205, pretty sure you also need to indent line 210 by two spaces too.

Otherwise the path field has no parent value, ie the template: bit.

Please look at the gist, I have updated. Now error moves to line 14

Alright, I have fixed all the syntax issues. Still I cannot start the service so, I ran the following command to check if I can start the service without running it from console:

filebeat.exe -c filebeat.yml -e -v

here is the output

2016/05/02 08:41:21.859057 geolite.go:24: INFO GeoIP disabled: No paths were set
 under output.geoip.paths
2016/05/02 08:41:21.861010 outputs.go:126: INFO Activated elasticsearch as outpu
t plugin.
2016/05/02 08:41:21.861010 publish.go:288: INFO Publisher name: Files
2016/05/02 08:41:21.863940 async.go:78: INFO Flush Interval set to: 1s
2016/05/02 08:41:21.863940 async.go:84: INFO Max Bulk Size set to: 50
2016/05/02 08:41:21.863940 beat.go:147: INFO Init Beat: filebeat; Version: 1.2.1

2016/05/02 08:41:21.864916 beat.go:80: CRIT Config error: Error reading config f
ile: YAML config parsing failed on filebeat.yml: yaml: unmarshal errors:
  line 13: cannot unmarshal !!map into []config.ProspectorConfig. Exiting.

So, I have fixed the issues and updated the GIST. Now come to the original question.

After all that, still logs are not being shipped. This is what I can see in the logs:

Check file for harvesting
Update existing file for harvesting
Not harvesting, file didn't change
 End of file reached

Please, help!

It seems like the harvester reached the end of your file. To have a fresh start, remove the registry file and try again. But be aware, that this will start reading all files from scratch, but I assume that is what you intend to do.

@thyfere Opening new threads does not help to get a faster answer: Not harvesting, file didn't change

I still didn't resolve the issue. Filebeat is not reading the latest files in the folder. I did remove the registry but following are few of the last log entries:

2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-24.txt" (offset snapshot:0)
2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-24.txt
2016-06-02T10:13:07+03:00 DBG Update existing file for harvesting: C:\Serv-U Logs\Country-2015-Jun-29.txt
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jun-16.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jun-14.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jul-10.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-26.txt" (offset snapshot:0)
2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-26.txt
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-May-15.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-04.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-13.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-23.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-22.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jul-07.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-01.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-25.txt" (offset snapshot:0)
2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-25.txt
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Nov-03.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-27.txt" (offset snapshot:0)
2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-27.txt
2016-06-02T10:13:07+03:00 DBG Not harvesting, file didn't change: C:\Serv-U Logs\Country-2015-Jun-29.txt
2016-06-02T10:13:07+03:00 DBG Check file for harvesting: C:\Serv-U Logs\Country-2015-Jun-30.txt
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jul-15.txt; Backoff now.

Moreover, filebeat service stops automatically.

Can you share the lost log entries before filebeat shuts down?

These are last entries.

That is somehow strange as I would expect if Filebeat shuts down to see some shutdown messages. Could it be that something on your system kills filebeat?

This is the exact same issue I am having with Alpha 3, only my service doesn't stop...

filebeat-not-polling-the-directory