Filebeat not harvesting new files


(Kawika A Moss) #1

What is working?
If I create my custom logs with the ruby logger class and start filebeats, it will harvest and send the data to kibana.

This is not what we want with these logs, we are trying to spit out a log in json format, but in one line, for filebeats to push to ES and the Ruby logger adds a string to the beginning of each log, which we don't want..

What isn't working??
If I use the ruby file class to just write the log, in json format, the filebeats harvester does not recognize the log as a new log. The log says "File Didn't Change"

Example of the logs:

           ***Log that is not being harvested***

{:app_name=>"nuix_processingtest2", :event_name=>"end_batch", :server_name=>"kxmoss141121.consilio.com", :log_level=>"Metric", :message=>"nuix processingtest2 end_batch", :instance_id=>"4714", :app_version=>"6.2", :app_env=>"Dev", :elapsed_id=>"4714", :elapsed_event=>"end", :process_user=>"consilio\\kxmoss", :end_user=>"consilio\\kxmoss", :logging_object=>"logging.rb/build_event", :origin_timestamp=>"2016-05-31T13:54:30.539Z", "extended_properties"=>{"nuix_processingtest2"=>{:end_batch=>{:total_items=>1000, :total_bytes=>15589746889, :communication_items=>9865, :communication_bytes=>35790976, :audited_items=>4523, :audited_bytes=>155579632}}}}

           ***Log that is being harvested***

# Logfile created on 2016-05-31 08:42:20 -0500 by logger.rb/44203 I, [2016-05-31T08:42:20.908719 #15284] INFO -- : {:app_name=>"nuix_processing", :event_name=>"end_batch", :server_name=>"kxmoss141121.consilio.com", :log_level=>"Metric", :message=>"nuix processing end_batch", :instance_id=>"4712", :app_version=>"6.2", :app_env=>"Dev", :elapsed_id=>"4712", :elapsed_event=>"end", :process_user=>"consilio\\kxmoss", :end_user=>"consilio\\kxmoss", :logging_object=>"logging.rb/build_event", :origin_timestamp=>"2016-05-31T13:42:20.907Z", "extended_properties"=>{"nuix_processing"=>{:end_batch=>{:total_items=>2760, :total_bytes=>155578816, :communication_items=>1747, :communication_bytes=>35797666, :audited_items=>2701, :audited_bytes=>155578816}}}}

The part of the last log that isn't wanted is:
# Logfile created on 2016-05-31 08:42:20 -0500 by logger.rb/44203 I, [2016-05-31T08:42:20.908719 #15284] INFO -- :

My configuration for file beats...
################### Filebeat Configuration Example #########################

############################# Filebeat ######################################
filebeat:
  # List of prospectors to fetch data.
  prospectors:

    - input_type: log

      paths:
        - E:\Export\NUIX_LOGS\*.log
   
############################# Output ##########################################

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:
  kafka:
    hosts: ["[SERVERNAME]:9092"]
    topic: "logging_test"
    channel: "logging_test"
    worker: 1	

  ### Console output
  console:
    # Pretty print json event
    pretty: true

############################# Logging #########################################

logging:
  files:
    rotateeverybytes: 10485760 # = 10MB
  level: debug

(Kawika A Moss) #2

I figured out why beats wasn't grabbing the log created by the file class, it was because there was no newline at the end of the string.

NOTE: Logs must have a newline at the end for filebeats to harvest


(ruflin) #3

@kalapakim Glad you found the issue. Filebeat currently relies on the new line. There has been a feature request to change this: https://github.com/elastic/beats/issues/1324


(system) #4

This topic was automatically closed after 21 days. New replies are no longer allowed.