Parsing Dynatrace dashboard reports in XML format

Elastic Stack Logstash
4

Hi Badger,

There are some event processing errors in my ruby filter for some of my input files and i'm not able to detect the root cause, because there is no enough information even after i set "log.level: trace". I can not find out what is the difference btwn the successful ones and the failed ones

If you can teach me how to log from the loop inside the ruby script, i think i can figure out the source of the error. I've checked some other post and found an example of it but i didn't get the file-suffix thing.

The error is at line 25 as you can see below:

 [2019-11-13T15:02:11,610][ERROR][logstash.filters.ruby    ][main] Could not process event: no implicit conversion of String into Integer {:script_path=>"/home/xxx/code/dynatrace.rb", :class=>"TypeError", :backtrace=>["org/jruby/RubyArray.java:1483:in `[]'", "/home/xxx/code/dynatrace.rb:25:in `block in filter'", "org/jruby/RubyHash.java:1417:in `each'", "/home/xxx/code/dynatrace.rb:24:in `block in filter'", "org/jruby/RubyArray.java:1800:in `each'", "/home/xxx/code/dynatrace.rb:9:in `block in filter'", "org/jruby/RubyArray.java:1800:in `each'", "/home/xxx/code/dynatrace.rb:8:in `filter'", "/home/xxx/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby/script/context.rb:55:in `execute_filter'", "/home/xxx/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby/script.rb:30:in `execute'", "/home/xxx/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby.rb:98:in `file_script'", "/home/xxx/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby.rb:84:in `filter'", "/home/xxx/logstash-7.4.1/logstash-core/lib/logstash/filters/base.rb:143:in `do_filter'", "/home/xxx/logstash-7.4.1/logstash-core/lib/logstash/filters/base.rb:162:in `block in multi_filter'", "org/jruby/RubyArray.java:1800:in `each'", "/home/xxx/logstash-7.4.1/logstash-core/lib/logstash/filters/base.rb:159:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:115:in `multi_filter'", "/home/xxx/logstash-7.4.1/logstash-core/lib/logstash/java_pipeline.rb:243:in `block in start_workers'"]}
/home/xxx/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated

Logstash XML parsing output

I did some changes to your first version, the latest schema for the output event is the following:

{
    "chartdashlet" => "Banka Sigortacılığı Operasyon Adet ",
     "@timestamp" => 2019-11-13T11:27:03.550Z,
     "measure" => "SGT_GetirAktifTeklifPoliceMusteriNoTarihIle",
     "inputtype" => "dynatrace_dashboard_report",
     "@version" => "1",
     "measure_agg_type" => "Count",
     "value" => 2,
     "record_timestamp" => 2019-11-13T10:19:00.000Z,
     "dashboard" => "048_(SGT)_BANKA_SIGORTACILIĞI"
}

Final version of the script and the conf:

filter {
  xml {
    source => "message"
    target => "[@metadata][theXML]"
    force_array => false
    remove_field => [ "message" ]
    xpath => [ "/dashboardreport/source/filters/filter/text()" , "[@metadata][report_filter_text]" ]
  }
  dissect {
    mapping => { "[@metadata][report_filter_text]" => "%{?filter_type}?%{from_millis}:%{?end_millis}" }
  }
  date {
    match => [ "from_millis", "UNIX_MS" ]
    target => "record_timestamp"
  }
  ruby {
    path => "/home/xxx/code/dynatrace.rb"
  }
}
output {
  stdout {
    codec => rubydebug { metadata => true }
  }
 
}

I've did these changes in order to set the original record timestamp inside the xml file, and i get it by parsing the "filter" node inside the xml file and i added dashboard name to each record.

def register(params)
end

def filter(event)
    theEvents = []
    xml =  event.get("[@metadata][theXML]")
    recordTimestamp = event.get("record_timestamp")
    xml["data"]["chartdashlet"].each { |dashlet|
        dashlet["measures"]["measure"].each { |y|
            measurement = y["measurement"]
            anEvent = Hash[ "inputtype", "dynatrace_dashboard_report", "dashboard", xml["name"], "chartdashlet", dashlet["name"], "measure", y["measure"], "measure_agg_type", y["aggregation"] ]
            anEvent["record_timestamp"] = recordTimestamp
            if y["aggregation"] == "Average"
                sum = 0.0
                count = 0
                measurement.each { |z|
                    sum += z["sum"].to_f
                    count += z["count"].to_i
                }
                anEvent["value"] = sum/count
                theEvents << LogStash::Event.new(anEvent)
            elsif y["aggregation"] == "Count"
                count = 0
                measurement.each { |z|
                    count += z["count"].to_i
                }
                anEvent["value"] = count
                theEvents << LogStash::Event.new(anEvent)
            end
        }
    }
    theEvents
end

Best,
Volkan