Ruby exception occurred: undefined method `[]' for nil:NilClass

[Hanneu]

Hello experts,
I have been getting very many below error and warning messages.

1.) [ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `[]' for nil:NilClass

2.) ][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"ind_ecs_airs_counter-2018.07.18", :_type=>"logs", :_routing=>nil}, 2018-07-18T04:01:01.541Z POAIR001 DATE/TIME,EMAP Request V1,EMAP Response V1,EMAP Request V2,EMAP Response V2,ETSI MAP Request V2,ETSI MAP Response V2,ETSI MAP Request V3,ETSI MAP Response V3], :response=>{"index"=>{"_index"=>"ind_ecs_airs_counter-2018.07.18", "_type"=>"logs", "_id"=>"AWSrjIbB6iFc8xSmLrcS", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [ETSI_MAP_Response_V3]", "caused_by"=>{"type"=>"number_format_exception", "reason"=>"For input string: "ETSI MAP Response V3""}}}}}

Can someone please suggest?

[Badger]

For the first one, you need to show us at least the ruby filter from your logstash configuration.

For the second, you have a field that contains "ETSI MAP Response V3" and elasticsearch is expecting that field to be a number.

[Hanneu]

I am looking for ruby filter in files under logstash config (/etc/logstash/conf.d). However, I am not 100% sure how to check for ruby filter. Can you suggest?

Second issue, I understand elastic search is expecting the field "EPSI Map Response V3" to be in numeric format. However, is there a way we can change to string or any other non-numeric type?

[Hanneu]

For first issue, would the ruby filter be under /usr/share/logstash/logstash-core/lib/logstash/filters?

[Badger]

No, I would expect it to be in your logstash config (/etc/logstash/conf.d or somewhere like that).

[Hanneu]

ruby
{
code => " event.set('TotalMap', event.get('EMAP_Request_V1').to_i + event.get('EMAP_Request_V2').to_i + event.get('ETSI_MAP_Request_V2').to_i + event.get('ETSI_MAP_Request_V3').to_i )
event.set('SuccessMap', event.get('EMAP_Response_V1').to_i + event.get('EMAP_Response_V2').to_i + event.get('ETSI_MAP_Response_V2').to_i + event.get('ETSI_MAP_Response_V3').to_i ) "
}

ruby
{
code => " if (event.get('BalanceEnquiryOut').to_i + event.get('AccumulatorEnquiryOut').to_i != 0)
event.set('balance', ( event.get('BalanceEnquiryIn').to_i + event.get('AccumulatorEnquiryIn').to_i ) / ( event.get('BalanceEnquiryOut').to_i + event.get('AccumulatorEnquiryOut').to_i ))
else
event.set('balance', 0 )
end "
}

[Badger]

That's the kind of thing we are looking for. Although I would not expect those to produce that error. The NilClass could be produced by doing event.get on a field that does not exist. ruby seems to be OK doing to_i on nil, so those filters should be OK. I'd be looking for something that indexes into an array without checking for nil first. A reference to something like

event.get("noSuchField")[0]

[Hanneu]

ruby
{
code => "
event.to_hash.keys.each { |k|
if k.start_with? '/Air:GetFaFList'
event.set('GetFaFList',event.get('GetFaFList').to_i + event.get(k).to_i)

      elsif k.start_with? '/Air:UpdateFaFList'
        event.set('UpdateFaFList',event.get('UpdateFaFList').to_i + event.get(k).to_i)

[Badger]

k is certain to exist, so the start_with? should be OK, and the only other thing that one does it to_i, which we know is OK.