How to exclude XML & json key-value if key length is greater than 15 char and value length is greater than 100 char

Elastic Stack Logstash
8

OK, so you want to recursively remove long keys/values from your event. That will require a ruby function. That is going to be similar to this.

    xml { source => "message" target => "xml" force_array => false }
    ruby {
        code => '
            def removeBigThings(object, name, event)
                if object
                    if object.kind_of?(Hash) and object != {}
                        object.each { |k, v| removeBigThings(v, "#{name}[#{k}]", event) }
                    elsif object.kind_of?(Array) and object != []
                        object.each_index { |i|
                            removeBigThings(object[i], "#{name}[#{i}]", event)
                        }
                    else
                        lastElement = name.gsub(/^.*\[/, "").gsub(/\]$/, "")
                        if lastElement.length > 15 or object.to_s.length > 100
                            event.remove(name)
                        end
                    end
                end
            end

            event.to_hash.each { |k, v|
                removeBigThings(v, "[#{k}]", event)
            }
        '
    }

Note that I have used force_array => false. You do not have to, I just do not like everything being an array with one member.

Note also that I used "or" in the test lastElement.length > 15 or object.to_s.length > 100. You said "and" but that would result in nothing being removed.

Note also that this will delete [message], since that is more than 100 characters long. If you only want to modify things in [xml] you could change

            event.to_hash.each { |k, v|
                removeBigThings(v, "[#{k}]", event)
            }

to

removeBigThings(event.get("xml"), "[xml]", event)
1 Like