Shipping logs from multiple files


(Thy Fere) #1

Hi,

I am using ELK stack on Windows box and configuring filebeat to ship logs from a computer's folder where logs have been generated everyday on a new file. How can I configure .yml conf to ship these logs to Elasticsearch?

Thanks.


(Mark Walkom) #2

If you have specific questions I am sure we can help, but have you read the docs? https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html


(Thy Fere) #3

Yes, I read it but still I can't turn the service on. Here are my configurations:

filebeat:
  prospectors:

      paths:
        - C:\Serv-U Logs\*\*

      input_type: log

      force_close_files: true

  registry_file: "C:/ProgramData/filebeat/registry"

output:

  elasticsearch:

    hosts: ["IP Address"]

    index: "filebeat"



      path: "C:/ProgramData/filebeat/filebeat.template.json"


shipper:

  tags: ["tamuq-files", "serv-u"]

logging:

  to_files: true

  files:

    path: c:\programdata\filebeat\logs

    rotateeverybytes: 10485760 # = 10MB

    keepfiles: 7

(Mark Walkom) #4

What do you mean by this?


(Thy Fere) #5

I have installed the service but can't turn it on. It says:

Loading config file error: YAML config parsing failed on filebeat.yml: yaml: line 210: did not find expected key. Exiting.

On line 210 in .yml, the following configuration exists:

# Path to template file
      path: "C:/ProgramData/filebeat/filebeat.template.json"

(Mark Walkom) #6

Yaml is sensitive, make sure your indentations are correct.


(Thy Fere) #7

Really? How can I make sure? Is there any guideline?


(Mark Walkom) #8

Can you post your entire config somewhere - gist/pastebin/etc?


(Thy Fere) #9

Here you go:


(Mark Walkom) #10

Pretty sure your issue is here - https://gist.github.com/najamss/8382b64cf403d0eb00901162690469ac#file-filebeat-yml-L205-L210


(Thy Fere) #11

I am following this document but still issue is there... It's frustrating...

http://ess.khhq.net/wiki/YAML_Tutorial


(Mark Walkom) #12

You need to uncomment line 205, pretty sure you also need to indent line 210 by two spaces too.

Otherwise the path field has no parent value, ie the template: bit.


(Thy Fere) #13

Please look at the gist, I have updated. Now error moves to line 14


(Thy Fere) #14

Alright, I have fixed all the syntax issues. Still I cannot start the service so, I ran the following command to check if I can start the service without running it from console:

filebeat.exe -c filebeat.yml -e -v

here is the output

2016/05/02 08:41:21.859057 geolite.go:24: INFO GeoIP disabled: No paths were set
 under output.geoip.paths
2016/05/02 08:41:21.861010 outputs.go:126: INFO Activated elasticsearch as outpu
t plugin.
2016/05/02 08:41:21.861010 publish.go:288: INFO Publisher name: Files
2016/05/02 08:41:21.863940 async.go:78: INFO Flush Interval set to: 1s
2016/05/02 08:41:21.863940 async.go:84: INFO Max Bulk Size set to: 50
2016/05/02 08:41:21.863940 beat.go:147: INFO Init Beat: filebeat; Version: 1.2.1

2016/05/02 08:41:21.864916 beat.go:80: CRIT Config error: Error reading config f
ile: YAML config parsing failed on filebeat.yml: yaml: unmarshal errors:
  line 13: cannot unmarshal !!map into []config.ProspectorConfig. Exiting.

(Thy Fere) #15

So, I have fixed the issues and updated the GIST. Now come to the original question.


(Thy Fere) #16

After all that, still logs are not being shipped. This is what I can see in the logs:

Check file for harvesting
Update existing file for harvesting
Not harvesting, file didn't change
 End of file reached

Please, help!


(ruflin) #17

It seems like the harvester reached the end of your file. To have a fresh start, remove the registry file and try again. But be aware, that this will start reading all files from scratch, but I assume that is what you intend to do.


(ruflin) #18

@thyfere Opening new threads does not help to get a faster answer: Not harvesting, file didn't change


(Thy Fere) #19

I still didn't resolve the issue. Filebeat is not reading the latest files in the folder. I did remove the registry but following are few of the last log entries:

2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-24.txt" (offset snapshot:0)
2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-24.txt
2016-06-02T10:13:07+03:00 DBG Update existing file for harvesting: C:\Serv-U Logs\Country-2015-Jun-29.txt
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jun-16.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jun-14.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jul-10.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-26.txt" (offset snapshot:0)
2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-26.txt
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-May-15.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-04.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-13.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-23.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-22.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jul-07.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Mar-01.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-25.txt" (offset snapshot:0)
2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-25.txt
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Nov-03.txt; Backoff now.
2016-06-02T10:13:07+03:00 DBG harvest: "C:\Serv-U Logs\Country-2015-Nov-27.txt" (offset snapshot:0)
2016-06-02T10:13:07+03:00 INFO Harvester started for file: C:\Serv-U Logs\Country-2015-Nov-27.txt
2016-06-02T10:13:07+03:00 DBG Not harvesting, file didn't change: C:\Serv-U Logs\Country-2015-Jun-29.txt
2016-06-02T10:13:07+03:00 DBG Check file for harvesting: C:\Serv-U Logs\Country-2015-Jun-30.txt
2016-06-02T10:13:07+03:00 DBG End of file reached: C:\Serv-U Logs\Country-2015-Jul-15.txt; Backoff now.


(Thy Fere) #20

Moreover, filebeat service stops automatically.