Wildcards in logstash remove_field

[Gauti (Gautham)]

Hi All,

I have a data source with almost 692 fields, out of which only 200 fields are valid, i want to remove those fields , i tried using below one, but no luck

mutate { remove_field => [ ".value" ] }

all the unwanted fields are ending with ".value" filed , is there any way i can achieve this?

Few example fields FYI..
inc_active.value
inc_additional_assignee_list.value
inc_approval.value
inc_assigned_to.value
inc_close_notes.value

Any advice please.

Thanks
Gautham

[Badger]

If they are top level fields you can use a prune filter with a wildcard in blacklist_names. If those examples are actually top-level objects that contain a value field then you will have to do it in ruby.

[Gauti (Gautham)]

@Badger i tried to replicate the below ruby code,

How to remove subfield using ruby

Hi , I am using ruby filter to remove fields that start with 'ID'. In the same way how can i remove subfield of a field called 'outputdata'. Below, code couldn't remove. ruby { code => " event.to_hash.keys.each { |k| event.remove(k) if k.start_with?('outputdata.ID:') } Thank you

I used something like below and was getting error,

ruby { code => 'event['result'].keys.each { |k| event['result'].remove(k) if k.end_with?('.value')' }

can you provide some ruby code example for my scenario.

Thanks
Gautham

[Badger]

You need to explain exactly what you want to do. When you say inc_active.value, is inc_active a top level object that contains a field called value? If so, do you want to remove the value field or the inc_active object? Or is inc_active.value a top level field that has a period in its name?

[Gauti (Gautham)]

@Badger I'm actually pulling data from servicenow this has duplicate fields like

 inc_active.value : true
 inc_active.display_value : true

inc_additional_assignee_list.value : windows
inc_additional_assignee_list.display_value : windows

inc_approval.value : 1
inc_approval.display_value : 1

inc_assigned_to.value : 0
inc_assigned_to.display_value : 0

inc_close_notes.value : NA
inc_close_notes.display_value : NA

From these documents i need to remove all fields which is ending with ".value", so that i my documents in elk will have only display_value fields

 inc_active.display_value : true
inc_additional_assignee_list.display_value : windows
inc_approval.display_value : 1
inc_assigned_to.display_value : 0
inc_close_notes.display_value : NA

These are few fields, like this i have a total of 200 fields to be removed. Instead of adding all these 200 fields in prune blacklist, was checking if we can put a wildcard

Thanks
Gautham

[Badger]

Once again, is inc_approval.value a top level field, or is inc_approval an object that contains fields called value and display_value? If you do not understand the difference then add

output { stdout { codec => rubydebug } }

to your configuration and show us what the inc_approval fields look like.

[Gauti (Gautham)]

@Badger Its an object that contains the fields.

here is the rubydebug output

{
    "@timestamp" => 2020-07-17T18:29:01.629Z,
      "@version" => "1",
        "result" => {
                                  "inc_active" => {
            "display_value" => "true",
                    "value" => "true"
        },
                            "inc_additional_assignee_list" => {
            "display_value" => "windows",
                    "value" => "windows"
        },
              "inc_approval.display_value" => {
            "display_value" => "1",
                    "value" => "1"
        },
                                "inc_due_date" => {
            "display_value" => "0",
                    "value" => "0"
        },
                            "inc_close_notes" => {
            "display_value" => " NA",
                     "link" => "https://prod.service-now.com/api/now/table/7bd1900ae45",
                    "value" => "NA"
        },
              "inc_u_remove_device_from_local" => {
            "display_value" => nil,
                    "value" => ""
        },
                               "inc_u_subject" => {
            "display_value" => "High Visibility Notification OPEN Problem 663: Failure rate increase on Database ser",
                    "value" => "High Visibility Notification OPEN Problem 663: Failure rate increase on Database ser"
        },
                            "inc_approval_set" => {
            "display_value" => "",
                    "value" => ""
        },

[Badger]

OK, so you will need ruby. I have not tested it, but try

ruby {
    code => '
        event.to_hash.each { |k, v|
            if v.is_a? Hash and v.key? "value"
                event.remove("[#{k}][value]")
            end
        }
    '
}

[Gauti (Gautham)]

@Badger It didnt work still getting the same output.

[Badger]

I misread the rubydebug. How about this?

ruby {
    code => '
        event.get("result").each { |k, v|
            if v.is_a? Hash and v.key? "value"
                event.remove("[result][#{k}][value]")
            end
        }
    '
}

[Gauti (Gautham)]

@Badger Bingo.....It worked... Thank you very much.

A small query, can i add any exception to this, like whitelisting one or two fields without removing?

Thanks
Gautham

[Badger]

@Gauti

can i add any exception to this, like whitelisting one or two fields without removing?

You could do something like

unless [ "someKey", "anotherKey" ].include? k
    event.remove("[result][#{k}][value]")
end

If you have more than a couple of values to whitelist you might want to use a hash rather than an array for performance reasons.