Can anyone tell me if the init configuration option is supported and/or necessary in ruby script files, and if so, how to use it?
My script file (yesterday.rb) looks like this:
def register(params)
@f_timestamp = params['f_timestamp']
@tsh = params['tsh']
@tsm = params['tsm']
@tss = params['tss']
@ms = params['ms']
@file_hour = params['file_hour']
end
def filter(event)
date_obj = DateTime.parse(event.get(@f_timestamp))
tsh = event.get(@tsh)
tsm = event.get(@tsm)
tss = event.get(@tss)
ms = event.get(@ms)
file_hour = event.get(@file_hour)
if file_hour == '00' and tsh == '23'
event.set('f_timestamp',date_obj.year.to_s+' '+date_obj.month.to_s+' '+date_obj.prev_day.day.to_s+' '+tsh+':'+tsm+':'+tss+' '+ms)
else
event.set('f_timestamp',date_obj.year.to_s+' '+date_obj.month.to_s+' '+date_obj.day.to_s+' '+tsh+':'+tsm+':'+tss+' '+ms)
end
return [event]
end
Here's the filter from my logstash pipeline:
filter {
grok {
match => [ "source", "%{GREEDYDATA}/HMS.log_(?<file_year>\d{4})(?<file_month>\d{2})(?<file_day>\d{2})(?<file_hour>\d{2})" ]
}
grok {
match => [
"message", "^<(?<tsh>\d{2}):(?<tsm>\d{2}):(?<tss>\d{2}).(?<ms>\d{3}) %{NOTSPACE:sev} %{NOTSPACE:subsys} %{NOTSPACE:f1} %{DATA:f2}>%{GREEDYDATA:message}$"
]
overwrite => ["message"]
}
mutate {
add_field => {
"f_timestamp" => "%{file_year}-%{file_month}-%{file_day} %{tsh}:%{tsm}:%{tss}.%{ms}"
}
}
ruby {
id => "ruby"
path => "/usr/share/logstash/pipeline/yesterday.rb"
script_params => {
"f_timestamp" => "f_timestamp"
"tsh" => "tsh"
"tsm" => "tsm"
"tss" => "tss"
"ms" => "ms"
"file_hour" => "file_hour"
}
}
date {
match => ["f_timestamp","yyyy MM dd HH:mm:ss SSS"]
timezone => "UTC"
}
mutate {
remove_field => [ "t","file_year","file_month","file_day","file_hour","tsh","tsm","tss","ms" ]
}
}
Here are some sample messages from the log (/opt/servera/HMS.log_20180326022934.log) that's being ingested:
<02:19:07.304 *WRN* BUS 3246:3654 0:0>[handleKeepAliveTimer-dpDaemonMonitor.cpp-674]:Keep Alive Timer for [2, cmgrd] Received, DaemonState=DP_STATE_ACTIVE keepAliveRetryCount=1, keepAliveMaxRetryCount=3
<02:19:09.386 *WRN* CWS 4851:8265 0:0><CalRegMgr-R2-1212:0-294-220-1-0>[sendClearNotify(Cciber.cpp:844)] CWS_MSGTYPE_CLEAR_NOTIFY with ClearNotifyData:mCauseCode:0,mCode:98000
<02:19:10.188 *WRN* LAC 851:865 0:0><CcSpp-R6-1212:0-294-224-1-0>[init(SsReg.cpp:68)] Using default one. OptiCalIC
<02:19:10.136 *WRN* CWR 351:23665 0:0><oCsc-1212:0-274-1161-1-11894>[activatedTimeout(CscSm.cpp:1043)] Clear the buffer, type:BE, delay:0
Running these four messages through logstash (6.2.4) causes intermittent _rubyexceptions and _dateparsefailures. When I examine the f_timestamp of the failed messages they look like this:
? f_timestamp 2018-03-26 02:19:10.188
When the tags are missing the f_timestap looks like this (as expected):
?f_timestamp 2018 3 26 02:19:10 136