# Shipping logs from multiple files
**URL:** https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881
**Category:** Beats
**Tags:** filebeat
**Created:** [May 1, 2016, 7:31am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881 "2016-05-01T07:31:36Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [May 1, 2016, 7:31am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/1 "2016-05-01T07:31:36Z")
Hi,
I am using ELK stack on Windows box and configuring filebeat to ship logs from a computer's folder where logs have been generated everyday on a new file. How can I configure .yml conf to ship these logs to Elasticsearch?
Thanks.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 1, 2016, 10:29pm UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/2 "2016-05-01T22:29:00Z")
If you have specific questions I am sure we can help, but have you read the docs? [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html)
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [May 2, 2016, 5:23am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/3 "2016-05-02T05:23:05Z")
Yes, I read it but still I can't turn the service on. Here are my configurations:
```
filebeat:
prospectors:
paths:
- C:\Serv-U Logs\*\*
input_type: log
force_close_files: true
registry_file: "C:/ProgramData/filebeat/registry"
output:
elasticsearch:
hosts: ["IP Address"]
index: "filebeat"
path: "C:/ProgramData/filebeat/filebeat.template.json"
shipper:
tags: ["tamuq-files", "serv-u"]
logging:
to_files: true
files:
path: c:\programdata\filebeat\logs
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7
```
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 2, 2016, 5:24am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/4 "2016-05-02T05:24:09Z")
> [@thyfere](#):
>
> I can't turn the service on
What do you mean by this?
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [May 2, 2016, 5:27am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/5 "2016-05-02T05:27:30Z")
I have installed the service but can't turn it on. It says:
`Loading config file error: YAML config parsing failed on filebeat.yml: yaml: line 210: did not find expected key. Exiting.`
On line 210 in .yml, the following configuration exists:
```
# Path to template file
path: "C:/ProgramData/filebeat/filebeat.template.json"
```
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 2, 2016, 5:32am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/6 "2016-05-02T05:32:56Z")
Yaml is sensitive, make sure your indentations are correct.
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [May 2, 2016, 5:39am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/7 "2016-05-02T05:39:26Z")
Really? How can I make sure? Is there any guideline?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 2, 2016, 5:59am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/8 "2016-05-02T05:59:50Z")
Can you post your entire config somewhere - gist/pastebin/etc?
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [May 2, 2016, 6:59am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/9 "2016-05-02T06:59:44Z")
Here you go:
>
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 2, 2016, 7:05am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/10 "2016-05-02T07:05:11Z")
Pretty sure your issue is here - [https://gist.github.com/najamss/8382b64cf403d0eb00901162690469ac#file-filebeat-yml-L205-L210](https://gist.github.com/najamss/8382b64cf403d0eb00901162690469ac#file-filebeat-yml-L205-L210)
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [May 2, 2016, 7:50am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/11 "2016-05-02T07:50:42Z")
I am following this document but still issue is there... It's frustrating...
[http://ess.khhq.net/wiki/YAML\_Tutorial](http://ess.khhq.net/wiki/YAML_Tutorial)
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [May 2, 2016, 7:53am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/12 "2016-05-02T07:53:36Z")
You need to uncomment line 205, pretty sure you also need to indent line 210 by two spaces too.
Otherwise the `path` field has no parent value, ie the `template:` bit.
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [May 2, 2016, 7:56am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/13 "2016-05-02T07:56:03Z")
Please look at the gist, I have updated. Now error moves to line 14
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [May 2, 2016, 8:43am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/14 "2016-05-02T08:43:45Z")
Alright, I have fixed all the syntax issues. Still I cannot start the service so, I ran the following command to check if I can start the service without running it from console:
filebeat.exe -c filebeat.yml -e -v
here is the output
```
2016/05/02 08:41:21.859057 geolite.go:24: INFO GeoIP disabled: No paths were set
under output.geoip.paths
2016/05/02 08:41:21.861010 outputs.go:126: INFO Activated elasticsearch as outpu
t plugin.
2016/05/02 08:41:21.861010 publish.go:288: INFO Publisher name: Files
2016/05/02 08:41:21.863940 async.go:78: INFO Flush Interval set to: 1s
2016/05/02 08:41:21.863940 async.go:84: INFO Max Bulk Size set to: 50
2016/05/02 08:41:21.863940 beat.go:147: INFO Init Beat: filebeat; Version: 1.2.1
2016/05/02 08:41:21.864916 beat.go:80: CRIT Config error: Error reading config f
ile: YAML config parsing failed on filebeat.yml: yaml: unmarshal errors:
line 13: cannot unmarshal !!map into []config.ProspectorConfig. Exiting.
```
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [May 2, 2016, 11:56am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/15 "2016-05-02T11:56:31Z")
So, I have fixed the issues and updated the GIST. Now come to the original question.
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [May 3, 2016, 5:22am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/16 "2016-05-03T05:22:51Z")
After all that, still logs are not being shipped. This is what I can see in the logs:
```
Check file for harvesting
Update existing file for harvesting
Not harvesting, file didn't change
End of file reached
```
Please, help!
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [May 3, 2016, 2:54pm UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/17 "2016-05-03T14:54:40Z")
It seems like the harvester reached the end of your file. To have a fresh start, remove the registry file and try again. But be aware, that this will start reading all files from scratch, but I assume that is what you intend to do.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [May 3, 2016, 3:00pm UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/18 "2016-05-03T15:00:14Z")
@thyfere Opening new threads does not help to get a faster answer: [Not harvesting, file didn't change](https://discuss.elastic.co/t/not-harvesting-file-didnt-change/49021)
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [June 2, 2016, 7:19am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/19 "2016-06-02T07:19:51Z")
I still didn't resolve the issue. Filebeat is not reading the latest files in the folder. I did remove the registry but following are few of the last log entries:
> 2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-24.txt" (offset snapshot:0)
> 2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-24.txt
> 2016-06-02T10:13:07+03:00 DBG Update existing file for harvesting: C:\Serv-U Logs\Country-2015-Jun-29.txt
> 2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jun-16.txt; Backoff now.
> 2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jun-14.txt; Backoff now.
> 2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jul-10.txt; Backoff now.
> 2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-26.txt" (offset snapshot:0)
> 2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-26.txt
> 2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-May-15.txt; Backoff now.
> 2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-04.txt; Backoff now.
> 2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-13.txt; Backoff now.
> 2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-23.txt; Backoff now.
> 2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-22.txt; Backoff now.
> 2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jul-07.txt; Backoff now.
> 2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-01.txt; Backoff now.
> 2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-25.txt" (offset snapshot:0)
> 2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-25.txt
> 2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Nov-03.txt; Backoff now.
> 2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-27.txt" (offset snapshot:0)
> 2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-27.txt
> 2016-06-02T10:13:07+03:00 DBG Not harvesting, file didn't change: C:\Serv-U Logs\Country-2015-Jun-29.txt
> 2016-06-02T10:13:07+03:00 DBG Check file for harvesting: C:\Serv-U Logs\Country-2015-Jun-30.txt
> 2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jul-15.txt; Backoff now.
---
### Author:  [@thyfere](https://discuss.elastic.co/u/thyfere)
#### Post date: [June 2, 2016, 7:54am UTC](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881/20 "2016-06-02T07:54:05Z")
Moreover, filebeat service stops automatically.
[Next page](https://discuss.elastic.co/t/shipping-logs-from-multiple-files/48881.md?page=2)