Need help, filebeat doesn't seem working (beginner)


#1

Filebeat is not picking up logs and there is no log generated in C:\ProgramData\filebeat\Logs, any help is appreciated, thanks

filebeat:
  prospectors:
    -
      paths:
        - "D:/Input-log-files/SUNMRZBK5X942ZZ/*/*.log"
      input_type: log
      document_type: log
  registry_file: "C:/ProgramData/filebeat/registry"
  
output:
  elasticsearch:
    hosts: ["localhost:9200"]

shipper:
 
logging:
  to_files: true 
  files:  
    path: C:\ProgramData\filebeat\Logs
    name: filebeat
    rotateeverybytes: 10485760 # = 10MB

(ruflin) #2

Under logging, you can set the log level to info or debug: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html#_level This should provide you with more information on what is happening.


#3

Thank you, it looks like it is running but it is not showing on localhost:5601 Kibana, can this be index issue? I am using windows and I followed the guide to load the template with:

Invoke-WebRequest -Method Put -InFile filebeat.template.json -Uri http://localhost:9200/_template/filebeat?pretty

log after I changed info to debug:
2016-06-28T10:52:28-07:00 DBG Disable stderr logging
2016-06-28T10:52:28-07:00 DBG Initializing output plugins
2016-06-28T10:52:28-07:00 INFO GeoIP disabled: No paths were set under output.geoip.paths
2016-06-28T10:52:28-07:00 DBG ES Ping(url=http://localhost:9200, timeout=1m30s)
2016-06-28T10:52:28-07:00 DBG Ping status code: 200
2016-06-28T10:52:28-07:00 INFO Activated elasticsearch as output plugin.
2016-06-28T10:52:28-07:00 DBG Create output worker
2016-06-28T10:52:28-07:00 DBG No output is defined to store the topology. The server fields might not be filled.
2016-06-28T10:52:28-07:00 INFO Publisher name: WinlogBeat-test
2016-06-28T10:52:28-07:00 INFO Flush Interval set to: 1s
2016-06-28T10:52:28-07:00 INFO Max Bulk Size set to: 50
2016-06-28T10:52:28-07:00 DBG create bulk processing worker (interval=1s, bulk size=50)
2016-06-28T10:52:28-07:00 INFO Init Beat: filebeat; Version: 1.2.3
2016-06-28T10:52:28-07:00 INFO filebeat sucessfully setup. Start running.
2016-06-28T10:52:28-07:00 INFO Registry file set to: C:\ProgramData\filebeat\registry
2016-06-28T10:52:28-07:00 INFO Loading registrar data from C:\ProgramData\filebeat\registry
2016-06-28T10:52:28-07:00 DBG Set idleTimeoutDuration to 5s
2016-06-28T10:52:28-07:00 DBG File Configs: [/var/log/.log C:/ELK-Stack/filebeat-1.2.3-windows/log/.log]
2016-06-28T10:52:28-07:00 INFO Set ignore_older duration to 0
2016-06-28T10:52:28-07:00 INFO Set close_older duration to 1h0m0s
2016-06-28T10:52:28-07:00 INFO Set scan_frequency duration to 10s
2016-06-28T10:52:28-07:00 INFO Input type set to: log
2016-06-28T10:52:28-07:00 INFO Set backoff duration to 1s
2016-06-28T10:52:28-07:00 INFO Set max_backoff duration to 10s
2016-06-28T10:52:28-07:00 INFO force_close_file is disabled
2016-06-28T10:52:28-07:00 DBG Waiting for 1 prospectors to initialise
2016-06-28T10:52:28-07:00 INFO Starting prospector of type: log
2016-06-28T10:52:28-07:00 DBG exclude_files: []
2016-06-28T10:52:28-07:00 DBG scan path /var/log/.log
2016-06-28T10:52:28-07:00 DBG scan path C:/ELK-Stack/filebeat-1.2.3-windows/log/
.log
2016-06-28T10:52:28-07:00 DBG Check file for harvesting: C:\ELK-Stack\filebeat-1.2.3-windows\log\test1.log
2016-06-28T10:52:28-07:00 DBG Same file as before found. Fetch the state.
2016-06-28T10:52:28-07:00 DBG Check file for harvesting: C:\ELK-Stack\filebeat-1.2.3-windows\log\test2.log
2016-06-28T10:52:28-07:00 DBG Same file as before found. Fetch the state.
2016-06-28T10:52:28-07:00 DBG Check file for harvesting: C:\ELK-Stack\filebeat-1.2.3-windows\log\test3.log
2016-06-28T10:52:28-07:00 DBG Same file as before found. Fetch the state.
2016-06-28T10:52:28-07:00 DBG Check file for harvesting: C:\ELK-Stack\filebeat-1.2.3-windows\log\test4.log
2016-06-28T10:52:28-07:00 DBG Same file as before found. Fetch the state.
2016-06-28T10:52:28-07:00 DBG Check file for harvesting: C:\ELK-Stack\filebeat-1.2.3-windows\log\test5.log
2016-06-28T10:52:28-07:00 DBG Same file as before found. Fetch the state.
2016-06-28T10:52:28-07:00 DBG Check file for harvesting: C:\ELK-Stack\filebeat-1.2.3-windows\log\test6.log
2016-06-28T10:52:28-07:00 DBG Same file as before found. Fetch the state.
2016-06-28T10:52:28-07:00 DBG Check file for harvesting: C:\ELK-Stack\filebeat-1.2.3-windows\log\test7.log
2016-06-28T10:52:28-07:00 DBG Same file as before found. Fetch the state.
2016-06-28T10:52:28-07:00 DBG Check file for harvesting: C:\ELK-Stack\filebeat-1.2.3-windows\log\test8.log
2016-06-28T10:52:28-07:00 DBG Same file as before found. Fetch the state.
2016-06-28T10:52:28-07:00 DBG Check file for harvesting: C:\ELK-Stack\filebeat-1.2.3-windows\log\test9.log
2016-06-28T10:52:28-07:00 DBG Same file as before found. Fetch the state.
2016-06-28T10:52:28-07:00 DBG Start harvesting unknown file: C:\ELK-Stack\filebeat-1.2.3-windows\log\test8.log
2016-06-28T10:52:28-07:00 DBG Resuming harvester on a previously harvested file: C:\ELK-Stack\filebeat-1.2.3-windows\log\test8.log
2016-06-28T10:52:28-07:00 DBG Start harvesting unknown file: C:\ELK-Stack\filebeat-1.2.3-windows\log\test9.log
2016-06-28T10:52:28-07:00 DBG Resuming harvester on a previously harvested file: C:\ELK-Stack\filebeat-1.2.3-windows\log\test9.log
2016-06-28T10:52:28-07:00 DBG harvest: "C:\ELK-Stack\filebeat-1.2.3-windows\log\test9.log" (offset snapshot:0)



(ruflin) #4

It seems like you started filebeat previously and it has already read your log files. Filebeat continues reading at the last position. To restart from scratch remove the registry file (path can be found in your config). In case there were no new log files in the last 15 minutes, make sure to query a longer period in Kibana.


(system) #5

This topic was automatically closed after 21 days. New replies are no longer allowed.