Struggling with '-' into field as value

Elastic Stack Logstash
1

Greetings,

For very long time, I'm struggling with those errors into my logstash server:

[2023-07-25T17:13:15,009][WARN ][logstash.outputs.elasticsearch][5555_winlogbeat][413af53fed5d62fe27389e3c6e0cc4781e6d3cffc048c8a8d0249858b65de473] Could not index event to Elasticsearch. status: 400

The reason for that is because the source data contains an '-' as value and the field has a type of IP, so it will drop the event because of the type mismatch:
"reason"=>"failed to parse field [wifi.ap.ip] of type [ip] in document with id 'xe3mjokB6KEi429pJSv5'. Preview of field's value: '-'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"'-' is not an IP string literal."

I always tought, that when we see a value of '-' in Kibana, it was because the field was empty, but it the field seems to really contain an hyphen into it.

If there a way, using ruby or anything else to drop all fields that has an '-' as value? It could also only be the field that has '.ip' also. Storing a fields with '-' as its value is not very usefull into Kibana, so I won't be mad to drop them all!

Thank you all and best regards,
Yanick

2

Hello,

I was able to find this on the web:
(https://stackoverflow.com/questions/28957870/how-to-remove-all-fields-with-null-value-in-logstash-filter)

ruby {
  init => "
    def removeEmptyField(event,h,name)
      h.each do |k,v|
        if (v.is_a?(Hash) || v.is_a?(Array)) && v.to_s != '{}'
          removeEmptyField(event,v,String.new(name.to_s) << '[' << k.to_s << ']')
        else
        if v == '-' 
          event.remove(String.new(name.to_s) << '[' << k.to_s << ']')
        end
      end
    end
  end
  "
  code => "
    removeEmptyField event,event.to_hash,''
  "
}

However, now I'm trying the add a field called [debug][ruby] which will contain all removed fields by the ruby script. I made this change the the ruby code:

ruby {
  init => "
    def removeEmptyField(event,h,name)
      removed_fields = [ ]
      h.each do |k,v|
        if (v.is_a?(Hash) || v.is_a?(Array)) && v.to_s != '{}'
          removeEmptyField(event,v,String.new(name.to_s) << '[' << k.to_s << ']')
        else
        if v == '-' 
          removed_fields << 'name.to_s'
          event.remove(String.new(name.to_s) << '[' << k.to_s << ']')
        end
      end
      event.set('[debug][ruby]', removed_fields)
    end
  end
  "
  code => "
    removeEmptyField event,event.to_hash,''
  "
}

But the event.set doesn't seems to work; the field [debug][ruby] is never added.

If I change this line:
event.set('[debug][ruby]', removed_fields)
to:
event.set('[debug][ruby]', 'blablabla')

then I will see the field [debug][ruby] with the value 'blablabla'.

image

Maybe I miss something when working with an array.

Thanks all for your help!

Regards,
Yanick

3

When I tried to reproduce this it is added as an empty array. That is because you have defined a recursive function, and event.set is getting called once for each call to removeEmptyField, and the last one wins. If there are no "-" values at the top level of the event then the last call will set [ruby][debug] to an empty array. You could make @removed_fields an instance variable. Try

    ruby {
        init => '
            def removeEmptyField(event,h,name)
                h.each do |k,v|
                    if (v.is_a?(Hash) || v.is_a?(Array)) && v.to_s != "{}"
                        removeEmptyField(event,v,String.new(name.to_s) << "[" << k.to_s << "]")
                    else
                        if v == "-"
                            @removed_fields << "#{name}[#{k}]"
                            event.remove("#{name}[#{k}]")
                        end
                    end
                    event.set("[debug][ruby]", @removed_fields)
                end
            end
        '
        code => '
            @removed_fields = [ ]
            removeEmptyField event,event.to_hash,""
        '
    }
4

Hi @Badger,

Thank you for your reply. I tested it and it works! You're a genius, I tried several things all day long and it took you 5 minutes to see what's going on!

However, just before I see your reply I also see there was a recursive call of the function, so I did that:

ruby {
  init => '
    @removed_fields = String.new

    def resetVariable
      @removed_fields = String(nil)
    end

    def removeEmptyField(event,h,name)
      h.each do |k,v|
        if (v.is_a?(Hash) || v.is_a?(Array)) && v.to_s != "{}"
          removeEmptyField(event,v,String.new(name.to_s) << "[" << k.to_s << "]")
        else
        if v == "-" 
          record_id = event.get("[winlog][record_id]")
          logger.info("Removed field " << name.to_s << "[" <<  k.to_s << "] - Record id: " << record_id.to_s)
          @removed_fields = @removed_fields + name.to_s + "[" + k.to_s + "]"
          event.remove(String.new(name.to_s))
        end
      end
      event.set("[debug2][ruby]", @removed_fields)
      
    end
  end
  '
  code => '
    removeEmptyField event,event.to_hash,""
    resetVariable
  '
}

With all the tests I made, I tought the issue was having the variable @removed_fields as an array. So I created another variable as a string ([debug2][ruby]).

The result was this:

image
image746×127 13.2 KB

While the result using the array into your solution is:

image
image758×146 16.3 KB

In both case the display is not looking good in Kibana. I'm wondering if where is a way at add line feed after each variable. I tried \n, but Kibana will show the \n in the output.

Using Winlogbeat, the Windows Event Viewer will send fields witl carriage return in it and Kibana will display it correctly:
image

So thank you very much for your help. Will mark your provided code as "Solution",

Best Regards,
Yanick

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.