I am trying to use a big logstash config file to parse my (atleast 100) csv file depending on the index created by filebeat.
So, I am using the following code within my logstash conf file for these particular csv files.
logstash::configfile { 'stats_filter':
content => '
filter {
if [type] == "stats" {
csv{
separator => ","
columns => ["Record Type","Record Code","Broker Name","Broker UUID","JVM Name","JVM UUID","Flow Name","Flow UUID","Application Name","Application UUID","Library Name","Library UUID","Record Start Date","Record Start Time","Record GMT Start Timestamp","Record End Date","Record End Time","Record GMT End Timestamp","Total Elapsed Time","Average Elapsed Time","Maximum Elapsed Time","Minimum Elapsed Time","Total CPU Time","Average CPU Time","Maximum CPU Time","Minimum CPU Time","CPU Time Waiting for Input Messages","Elapsed Time Waiting for Input Messages","Total Number of Input Messages","Total Size of Input Messages","Average Size of Input Messages","Maximum Size of Input Messages","Minimum Size of Input Messages","Number of Threads in Pool","Time Maximum Number of Threads reached","Total Number of Errors","Total Number of Messages with Errors","Total Number of Errors Processing Messages","Total Number of Time Outs Waiting for Replies to Aggregate Messages","Total Number of Commits","Total Number of Backouts","Accounting Origin"]
}
ruby {
code => "
wanted_fields = ['Broker Name','JVM Name','Flow Name','Total CPU Time','Total Number of Input Messages','Record End Date','Record End Time']
event.to_hash.keys.each { |k|
event.remove(k) unless wanted_fields.include? k
}"
}
mutate {
rename => { "Broker Name" => broker_name }
rename => { "JVM Name" => jvm_name }
rename => { "Flow Name" => flowname }
rename => { "Total CPU Time" => cputime }
rename => { "Total Number of Input Messages" => input_messages }
add_field => {
"timestamp" => "%{Record End Date} %{Record End Time}"}
remove_field => ["Record End Date"]
remove_field => ["Record End Time"]
}
date{
match => ["timestamp","yyyy-MM-dd HH:mm:ss.SSSSSS"]
timezone => "Etc/UTC"
remove_field => ["timestamp"]
}
}
}
'
}
I am writing the output to elasticsearch server. But when using the above code, since the filter content is within single quotes, i am erroring out at the wanted_fields line since i am using a single quote there too. Is there anyway to make this work? I tried escaping the single quotes, use other options( like double quotes etc) but that gave me clear errors that I have to use single quotes for wanted_fields. Is there another option? The filter has to be within content since as i said before this is part of a really large logstash conf file and this has to be done this way for consistency.