Logstash Ruby exceptions

Elastic Stack Logstash
1

Hello together,

I have the following Logstash pipeline:

input {
   file {
  path => "/test/home.w10_vinfo-sg-war.log.*"
  start_position => "beginning"
  exclude => "*.gz"
  exclude => "*.snappy"
}
}
filter{
xml {
 store_xml => false
source => "message"
xpath => [
     "/log/@level", "level",
     "/log/msg/text()","msg_txt"
     ]
  }
  grok{ 
   break_on_match => false
   match => ["msg_txt", "Clat=\[(?<Clat>-?\d+)\]"]
   match => ["msg_txt", "Clon=\[(?<Clon>-?\d+)\]"]
   match => ["msg_txt", "Dlat=\[(?<Dlat>-?\d+)\]"]
  match => ["msg_txt", "Dlon=\[(?<Dlon>-?\d+)\]"]
  }
  mutate {
   convert => { 
      "Clat" => "float"
      "Clon" => "float"
      "Dlat" => "float"
      "Dlon" => "float"
      }
     }
    if [Clat] != '' {
   ruby {
   code => "
        if !event['Clat'].to_s.include? '.'
        event['Clat'] = event['Clat'] * 360 / (4294967296)
        end
      "
     }
    }
  if [Clon] != '' {
  ruby {
   code => "
        if !event['Clon'].to_s.include? '.'
        event['Clon'] = event['Clon'] * 360 / (4294967296)
        end
      "
   }
  }
  if [Dlat] != '' {
ruby {
code => "
        if !event['Dlat'].to_s.include? '.'
        event['Dlat'] = event['Dlat'] * 360 / (4294967296)
        end
      "
  }
  }
  if [Dlon] != '' {
 ruby {
  code => "
        if !event['Dlon'].to_s.include? '.'
        event['Dlon'] = event['Dlon'] * 360 / (4294967296)
        end
      "
  }
  }    
ruby {
code => "
        event['Clat'] = event['Clat'].round(2)
        event['Clon'] = event['Clon'].round(2)
        event['Dlat'] = event['Dlat'].round(2)
        event['Dlon'] = event['Dlon'].round(2)
      "
  }
  mutate {    
add_field => [ "[location]", "%{Clon}" ]
add_field => [ "[location]", "%{Clat}" ]
 convert => [ "[location]", "float" ]
 }
 }
output {
  elasticsearch {
     hosts => "localhost:9200"
  }
stdout{}
}

And the correpsonding log file:
<log level="INFO" time="Tue Sep 08 11:42:39 EDT 2015" timel="1441726959272" id="1234567890" cat="COMMUNICATION" comp="WEB" host="localhost" req="" app="" usr="" thread="" origin=""><msg><![CDATA[Method=GET URL=http://test:80/testus?OP=gtm&TReq(Clat=[429566997], Clon=[-1372987576], Decoding_Feat=[], Dlat=[0], Dlon=[0], Accept-Encoding=gzip, Accept=*/*) Result(Content-Encoding=[gzip], Content-Length=[2815], ntCoent-Length=[5276], Content-Type=[text/xml; charset=utf-8]) Status=200 Times=TISP:344/CSI:-/Me:0/Total:344]]></msg><info></info><excp></excp></log>

However, the xml, grok and mutate parts are working for sure. Unfortunately, the ruby part causes problems because it does not round, as intended, the numbers to two decimals. Therefore, location cannot be used. I get the following snippet error message:
"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"illegal latitude value [3.91861529E8] for location"}}}}, :level=>:warn}
And two ruby exceptions:
Ruby exception occurred: undefined method "*" for nil:NilClass {:level=>:error} Ruby exception occurred: undefined method "round" for nil:NilClass {:level=>:error}
I already tried to search for relevant problems in Google, but not very successful.

Do you maybe have suggestions where the problem can be?

Thanks in advance.

2

Your fields event['*'] are nil when you used * and round. Can you guess why?

3

No sorry, why this problem occurs? Could you please explain?

4

My guess is it's breaking somewhere in the XML or grok filter. Why don't check for nil before entering the ruby filter or before entering the ruby loop. That will tell you.

5

Hi,

I changed to code, now I get the following message: "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"illegal latitude value [4.99537813E8] for location"}}}}, :level=>:warn}

mutate {
   convert => { 
      "Clat" => "float"
      "Clon" => "float"
      "Dlat" => "float"
      "Dlon" => "float"
}
  }
  if [Clat] != '' {
 ruby {
 code => "
        if !event['Clat'].to_s.include? '.'
        event['Clat'] = (event['Clat'].to_f * 360 / (4294967296)).round(2)
        end
      "
 }
 }
  if [Clon] != '' {
  ruby {
code => "
        if !event['Clon'].to_s.include? '.'
        event['Clon'] = (event['Clon'].to_f * 360 / (4294967296)).round(2)
        end
      "
  }
  }
  if [Dlat] != '' {
    ruby {
 code => "
        if !event['Dlat'].to_s.include? '.'
        event['Dlat'] = (event['Dlat'].to_f * 360 / (4294967296)).round(2)
        end
      "
 }
  }
  if [Dlon] != '' {
ruby {
code => "
        if !event['Dlon'].to_s.include? '.'
        event['Dlon'] = (event['Dlon'].to_f * 360 / (4294967296)).round(2)
        end
      "
  }
  }    
ruby {
code => "
        event['Clat'] = event['Clat'].to_f.round(2)
        event['Clon'] = event['Clon'].to_f.round(2)
        event['Dlat'] = event['Dlat'].to_f.round(2)
        event['Dlon'] = event['Dlon'].to_f.round(2)
      "
  }
  mutate {    
add_field => [ "[location]", "%{Clon}" ]
add_field => [ "[location]", "%{Clat}" ]
convert => [ "[location]", "float" ]
}

}

What I do not understand is, although I am rounding up the values, it has more than 2 decimal places.