How to exclude key value pair from KV filter if Key size greater than 15 char

Elastic Stack Logstash
1

We have log message which contain key value pairs and we are using KV filter to parse these data.
We need to exclude key value pair from KV filter if Key size greater than 15 char.
Need to exclude https://www.google.com key from below log events.

username: abc
password: 123
https://www.google.com: 2

      kv { 
	  source => kvmessage
	  value_split => ":"
	}

Can any one suggest solution to exclude these key value pair?

2

The kv filter cannot do that for you, you would have to use a ruby filter. I have not tested it but something like

ruby {
    code => '
        event.to_hash.each { |k, v|
            if k.length > 15
                event.remove(k)
            end
        }
    '
}
1 Like
3

Can you please correct below code. not getting any error but still fields are created with char greater than 15.

else if "WEBAPPS" in [labels][kind] {
grok {
match => { "message" => "%{NUMBER:Year}-%{DATA:Month}-%{NOTSPACE:Day}\s*%{NOTSPACE:Hour}:%{NOTSPACE:Minute}:%{NOTSPACE:Second}\s*%{WORD:AppName}\s*%{WORD:Loglevel}\s%{GREEDYDATA:kvmessage}" }
}

	ruby {
code => '
    event.to_hash.each { |k, v|
        if k.length > 15
            event.remove(k)
        end
    }
'
}
	kv { 
	  source => kvmessage
	  value_split => ":"
	}
  }
4

You cannot remove the fields if they do not exist. The ruby filter has to be after the kv filter that creates the fields.

5

@Badger ,
Thank you so much for your quick reply.
is it exclude particular key value pair or it exclude whole log line?
as per my log observation i am thinking it is dropping complete logline if he found key value is greater than 15. I need to drop that particular key value pair not complete log line

6

No, it just removes the key/value pair that has the long key.

7

@Badger ,
Yes it just removes key value pair. thank you so much for your help. it is working fine now. :slight_smile:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.