Hi Badger,
There are some event processing errors in my ruby filter for some of my input files and i'm not able to detect the root cause, because there is no enough information even after i set "log.level: trace". I can not find out what is the difference btwn the successful ones and the failed ones
If you can teach me how to log from the loop inside the ruby script, i think i can figure out the source of the error. I've checked some other post and found an example of it but i didn't get the file-suffix thing.
The error is at line 25 as you can see below:
[2019-11-13T15:02:11,610][ERROR][logstash.filters.ruby ][main] Could not process event: no implicit conversion of String into Integer {:script_path=>"/home/xxx/code/dynatrace.rb", :class=>"TypeError", :backtrace=>["org/jruby/RubyArray.java:1483:in `[]'", "/home/xxx/code/dynatrace.rb:25:in `block in filter'", "org/jruby/RubyHash.java:1417:in `each'", "/home/xxx/code/dynatrace.rb:24:in `block in filter'", "org/jruby/RubyArray.java:1800:in `each'", "/home/xxx/code/dynatrace.rb:9:in `block in filter'", "org/jruby/RubyArray.java:1800:in `each'", "/home/xxx/code/dynatrace.rb:8:in `filter'", "/home/xxx/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby/script/context.rb:55:in `execute_filter'", "/home/xxx/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby/script.rb:30:in `execute'", "/home/xxx/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby.rb:98:in `file_script'", "/home/xxx/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby.rb:84:in `filter'", "/home/xxx/logstash-7.4.1/logstash-core/lib/logstash/filters/base.rb:143:in `do_filter'", "/home/xxx/logstash-7.4.1/logstash-core/lib/logstash/filters/base.rb:162:in `block in multi_filter'", "org/jruby/RubyArray.java:1800:in `each'", "/home/xxx/logstash-7.4.1/logstash-core/lib/logstash/filters/base.rb:159:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:115:in `multi_filter'", "/home/xxx/logstash-7.4.1/logstash-core/lib/logstash/java_pipeline.rb:243:in `block in start_workers'"]}
/home/xxx/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
I did some changes to your first version, the latest schema for the output event is the following:
{
"chartdashlet" => "Banka Sigortacılığı Operasyon Adet ",
"@timestamp" => 2019-11-13T11:27:03.550Z,
"measure" => "SGT_GetirAktifTeklifPoliceMusteriNoTarihIle",
"inputtype" => "dynatrace_dashboard_report",
"@version" => "1",
"measure_agg_type" => "Count",
"value" => 2,
"record_timestamp" => 2019-11-13T10:19:00.000Z,
"dashboard" => "048_(SGT)_BANKA_SIGORTACILIĞI"
}
Final version of the script and the conf:
filter {
xml {
source => "message"
target => "[@metadata][theXML]"
force_array => false
remove_field => [ "message" ]
xpath => [ "/dashboardreport/source/filters/filter/text()" , "[@metadata][report_filter_text]" ]
}
dissect {
mapping => { "[@metadata][report_filter_text]" => "%{?filter_type}?%{from_millis}:%{?end_millis}" }
}
date {
match => [ "from_millis", "UNIX_MS" ]
target => "record_timestamp"
}
ruby {
path => "/home/xxx/code/dynatrace.rb"
}
}
output {
stdout {
codec => rubydebug { metadata => true }
}
}
I've did these changes in order to set the original record timestamp inside the xml file, and i get it by parsing the "filter" node inside the xml file and i added dashboard name to each record.
def register(params)
end
def filter(event)
theEvents = []
xml = event.get("[@metadata][theXML]")
recordTimestamp = event.get("record_timestamp")
xml["data"]["chartdashlet"].each { |dashlet|
dashlet["measures"]["measure"].each { |y|
measurement = y["measurement"]
anEvent = Hash[ "inputtype", "dynatrace_dashboard_report", "dashboard", xml["name"], "chartdashlet", dashlet["name"], "measure", y["measure"], "measure_agg_type", y["aggregation"] ]
anEvent["record_timestamp"] = recordTimestamp
if y["aggregation"] == "Average"
sum = 0.0
count = 0
measurement.each { |z|
sum += z["sum"].to_f
count += z["count"].to_i
}
anEvent["value"] = sum/count
theEvents << LogStash::Event.new(anEvent)
elsif y["aggregation"] == "Count"
count = 0
measurement.each { |z|
count += z["count"].to_i
}
anEvent["value"] = count
theEvents << LogStash::Event.new(anEvent)
end
}
}
theEvents
end
Best,
Volkan