What is working?
If I create my custom logs with the ruby logger class and start filebeats, it will harvest and send the data to kibana.
This is not what we want with these logs, we are trying to spit out a log in json format, but in one line, for filebeats to push to ES and the Ruby logger adds a string to the beginning of each log, which we don't want..
What isn't working??
If I use the ruby file class to just write the log, in json format, the filebeats harvester does not recognize the log as a new log. The log says "File Didn't Change"
Example of the logs:
***Log that is not being harvested***
{:app_name=>"nuix_processingtest2", :event_name=>"end_batch", :server_name=>"kxmoss141121.consilio.com", :log_level=>"Metric", :message=>"nuix processingtest2 end_batch", :instance_id=>"4714", :app_version=>"6.2", :app_env=>"Dev", :elapsed_id=>"4714", :elapsed_event=>"end", :process_user=>"consilio\\kxmoss", :end_user=>"consilio\\kxmoss", :logging_object=>"logging.rb/build_event", :origin_timestamp=>"2016-05-31T13:54:30.539Z", "extended_properties"=>{"nuix_processingtest2"=>{:end_batch=>{:total_items=>1000, :total_bytes=>15589746889, :communication_items=>9865, :communication_bytes=>35790976, :audited_items=>4523, :audited_bytes=>155579632}}}}
***Log that is being harvested***
# Logfile created on 2016-05-31 08:42:20 -0500 by logger.rb/44203 I, [2016-05-31T08:42:20.908719 #15284] INFO -- : {:app_name=>"nuix_processing", :event_name=>"end_batch", :server_name=>"kxmoss141121.consilio.com", :log_level=>"Metric", :message=>"nuix processing end_batch", :instance_id=>"4712", :app_version=>"6.2", :app_env=>"Dev", :elapsed_id=>"4712", :elapsed_event=>"end", :process_user=>"consilio\\kxmoss", :end_user=>"consilio\\kxmoss", :logging_object=>"logging.rb/build_event", :origin_timestamp=>"2016-05-31T13:42:20.907Z", "extended_properties"=>{"nuix_processing"=>{:end_batch=>{:total_items=>2760, :total_bytes=>155578816, :communication_items=>1747, :communication_bytes=>35797666, :audited_items=>2701, :audited_bytes=>155578816}}}}
The part of the last log that isn't wanted is:
# Logfile created on 2016-05-31 08:42:20 -0500 by logger.rb/44203 I, [2016-05-31T08:42:20.908719 #15284] INFO -- :
My configuration for file beats...
################### Filebeat Configuration Example #########################
############################# Filebeat ######################################
filebeat:
# List of prospectors to fetch data.
prospectors:
- input_type: log
paths:
- E:\Export\NUIX_LOGS\*.log
############################# Output ##########################################
# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:
kafka:
hosts: ["[SERVERNAME]:9092"]
topic: "logging_test"
channel: "logging_test"
worker: 1
### Console output
console:
# Pretty print json event
pretty: true
############################# Logging #########################################
logging:
files:
rotateeverybytes: 10485760 # = 10MB
level: debug