Filebeat harvesting issue


(Ramesh Narasimhan) #1

My filebeat configuration has been working fine. But it suddenly stopped working now. I have updated to the latest version of filebeat, restarted filebeat instance, restarted my application instance. but still no luck.filebeat; Version: 1.2.1
Filebeat log

 DBG  Disable stderr logging
 DBG  Initializing output plugins
 INFO GeoIP disabled: No paths were set under output.geoip.paths
 DBG  ES Ping(url=http://x.x.x.x:9200, timeout=1m30s)
 DBG  Ping status code: 200
 INFO Activated elasticsearch as output plugin.
 DBG  Create output worker
 DBG  No output is defined to store the topology. The server fields might not be filled.
 INFO Publisher name: ip-x-x-x-x
 INFO Flush Interval set to: 1s
 INFO Max Bulk Size set to: 50
 DBG  create bulk processing worker (interval=1s, bulk size=50)
 INFO Init Beat: filebeat; Version: 1.2.1
 INFO filebeat sucessfully setup. Start running.
 INFO Registry file set to: /var/lib/filebeat/registry
 INFO Loading registrar data from /var/lib/filebeat/registry
 DBG  Set idleTimeoutDuration to 5s
 DBG  File Configs: [/var/log/tomcat7/zkdcs.log]
 INFO Set ignore_older duration to 0
 INFO Set close_older duration to 1h0m0s
 INFO Set scan_frequency duration to 10s
 INFO Input type set to: log
 INFO Set backoff duration to 1s
 INFO Set max_backoff duration to 10s
 INFO force_close_file is disabled
 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
 DBG  Waiting for 1 prospectors to initialise
 INFO Starting prospector of type: log
 DBG  exclude_files: []
 DBG  scan path /var/log/tomcat7/zkdcs.log
 DBG  Check file for harvesting: /var/log/tomcat7/zkdcs.log
 DBG  Start harvesting unknown file: /var/log/tomcat7/zkdcs.log
 DBG  Same file as before found. Fetch the state and persist it.
 DBG  Resuming harvester on a previously harvested file: /var/log/tomcat7/zkdcs.log
 DBG  harvest: "/var/log/tomcat7/zkdcs.log" position:1073914596 (offset snapshot:0)
 INFO Harvester started for file: /var/log/tomcat7/zkdcs.log
 DBG  File was truncated as offset (%!s(int64=1073914596)) > size (%!s(int64=418066380)). Begin reading file from offset 0: /var/log/tomcat7/zkdcs.log
 INFO File was truncated. Begin reading file from offset 0: /var/log/tomcat7/zkdcs.log
 DBG  Registrar will re-save state for /var/log/tomcat7/zkdcs.log
 DBG  No pending prospectors. Finishing setup
 INFO All prospectors initialised with 2 states to persist
 INFO Starting Registrar
 INFO Start sending events to output
 DBG  scan path /var/log/tomcat7/zkdcs.log
 DBG  Check file for harvesting: /var/log/tomcat7/zkdcs.log
 DBG  Update existing file for harvesting: /var/log/tomcat7/zkdcs.log
 DBG  Not harvesting, file didn't change: /var/log/tomcat7/zkdcs.log
 DBG  Flushing spooler because of timeout. Events flushed: 0
DBG  Flushing spooler because of timeout. Events flushed: 0
 DBG  Start next scan
 DBG  scan path /var/log/tomcat7/zkdcs.log
 DBG  Check file for harvesting: /var/log/tomcat7/zkdcs.log
 DBG  Update existing file for harvesting: /var/log/tomcat7/zkdcs.log
 DBG  Not harvesting, file didn't change: /var/log/tomcat7/zkdcs.log
 DBG  Flushing spooler because of timeout. Events flushed: 0
 DBG  Start next scan
 DBG  scan path /var/log/tomcat7/zkdcs.log
 DBG  Check file for harvesting: /var/log/tomcat7/zkdcs.log
 DBG  Update existing file for harvesting: /var/log/tomcat7/zkdcs.log
 DBG  Not harvesting, file didn't change: /var/log/tomcat7/zkdcs.log
 DBG  Flushing spooler because of timeout. Events flushed: 0
 DBG  End of file reached: /var/log/tomcat7/zkdcs.log; Backoff now.

(ruflin) #2

Can you share some more details on what is not working anymore? Could you share your config file?


(system) #3