Hi,
I am using ELK stack on Windows box and configuring filebeat to ship logs from a computer's folder where logs have been generated everyday on a new file. How can I configure .yml conf to ship these logs to Elasticsearch?
Thanks.
If you have specific questions I am sure we can help, but have you read the docs? https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html
Yes, I read it but still I can't turn the service on. Here are my configurations:
filebeat:
prospectors:
paths:
- C:\Serv-U Logs\*\*
input_type: log
force_close_files: true
registry_file: "C:/ProgramData/filebeat/registry"
output:
elasticsearch:
hosts: ["IP Address"]
index: "filebeat"
path: "C:/ProgramData/filebeat/filebeat.template.json"
shipper:
tags: ["tamuq-files", "serv-u"]
logging:
to_files: true
files:
path: c:\programdata\filebeat\logs
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7What do you mean by this?
I have installed the service but can't turn it on. It says:
Loading config file error: YAML config parsing failed on filebeat.yml: yaml: line 210: did not find expected key. Exiting.
On line 210 in .yml, the following configuration exists:
# Path to template file
path: "C:/ProgramData/filebeat/filebeat.template.json"Yaml is sensitive, make sure your indentations are correct.
Really? How can I make sure? Is there any guideline?
Can you post your entire config somewhere - gist/pastebin/etc?
Here you go:
Pretty sure your issue is here - https://gist.github.com/najamss/8382b64cf403d0eb00901162690469ac#file-filebeat-yml-L205-L210
I am following this document but still issue is there... It's frustrating...
You need to uncomment line 205, pretty sure you also need to indent line 210 by two spaces too.
Otherwise the path field has no parent value, ie the template: bit.
Please look at the gist, I have updated. Now error moves to line 14
Alright, I have fixed all the syntax issues. Still I cannot start the service so, I ran the following command to check if I can start the service without running it from console:
filebeat.exe -c filebeat.yml -e -v
here is the output
2016/05/02 08:41:21.859057 geolite.go:24: INFO GeoIP disabled: No paths were set
under output.geoip.paths
2016/05/02 08:41:21.861010 outputs.go:126: INFO Activated elasticsearch as outpu
t plugin.
2016/05/02 08:41:21.861010 publish.go:288: INFO Publisher name: Files
2016/05/02 08:41:21.863940 async.go:78: INFO Flush Interval set to: 1s
2016/05/02 08:41:21.863940 async.go:84: INFO Max Bulk Size set to: 50
2016/05/02 08:41:21.863940 beat.go:147: INFO Init Beat: filebeat; Version: 1.2.1
2016/05/02 08:41:21.864916 beat.go:80: CRIT Config error: Error reading config f
ile: YAML config parsing failed on filebeat.yml: yaml: unmarshal errors:
line 13: cannot unmarshal !!map into []config.ProspectorConfig. Exiting.So, I have fixed the issues and updated the GIST. Now come to the original question.
After all that, still logs are not being shipped. This is what I can see in the logs:
Check file for harvesting
Update existing file for harvesting
Not harvesting, file didn't change
End of file reached
Please, help!
It seems like the harvester reached the end of your file. To have a fresh start, remove the registry file and try again. But be aware, that this will start reading all files from scratch, but I assume that is what you intend to do.
@thyfere Opening new threads does not help to get a faster answer: Not harvesting, file didn't change
I still didn't resolve the issue. Filebeat is not reading the latest files in the folder. I did remove the registry but following are few of the last log entries:
2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-24.txt" (offset snapshot:0)
2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-24.txt
2016-06-02T10:13:07+03:00 DBG Update existing file for harvesting: C:\Serv-U Logs\Country-2015-Jun-29.txt
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jun-16.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jun-14.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jul-10.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-26.txt" (offset snapshot:0)
2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-26.txt
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-May-15.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-04.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-13.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-23.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-22.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jul-07.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-01.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-25.txt" (offset snapshot:0)
2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-25.txt
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Nov-03.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-27.txt" (offset snapshot:0)
2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-27.txt
2016-06-02T10:13:07+03:00 DBG Not harvesting, file didn't change: C:\Serv-U Logs\Country-2015-Jun-29.txt
2016-06-02T10:13:07+03:00 DBG Check file for harvesting: C:\Serv-U Logs\Country-2015-Jun-30.txt
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jul-15.txt; Backoff now.
Moreover, filebeat service stops automatically.