To exclude around 350 fields in JSON in Logstash

Elastic Stack Logstash
6

OK, you will need to a ruby script that iterates over the event and makes recursive calls to itself for each hash and array that it finds. I have failed to write that a few times over the years, but to-day I got around to it...

Create a file called removeKeys.rb that contains

def register(params)
    @field = params['keys']
end

def removeKeys(object, name, keys, event)
#puts "removeKeys called for #{name}"
    if object
        if object.kind_of?(Hash) and object != {}
            object.each { |k, v| removeKeys(v, "#{name}[#{k}]", keys, event) }
        elsif object.kind_of?(Array) and object != []
            object.each_index { |i|
                removeKeys(object[i], "#{name}[#{i}]", keys, event)
            }
        else
            lastElement = name.gsub(/^.*\[/, "").gsub(/\]$/, "")
            if keys.include? lastElement
#puts "removing #{name}"
            event.remove(name)
            end
        end
    end
end

def filter(event)
    event.to_hash.each { |k, v|
            removeKeys(v, "[#{k}]", @field, event)
    }
    [event]
end

Then call it using

    ruby {
        path => "/home/user/removeKeys.rb"
        script_params => { keys => "events_url" }
    }

Note that keys can be an array

        script_params => { keys => [ "events_url", "comments" ] }

You probably need some error checking in the ruby code (you will know you do when logstash crashes) and my apologies if my ruby coding style makes your eyeballs bleed.

1 Like