Exclude keys

Elastic Stack Logstash
1

Input file:kvoptions.log
{"Name":"admin","ConfigurationObjectType":"ExecutionGroup"}

Configuration file:
input
{
file
{
path => "/home/murugar/Documents/kvoptions.log"
type => "file"
start_position => "beginning"
sincedb_path=> "/dev/null"

}
}
filter
{
grok
{
match => ["message","%{GREEDYDATA:msg}"]
}

    kv
    {
    source => "msg"
    field_split => ",\{\}"
    value_split => ":"
    exclude_keys => ["ConfigurationObjectType"]
    }

}

output
{

stdout
    {
    codec => rubydebug
  }

}

Obtained Ouput
{
"message" => "{"Name":"admin","ConfigurationObjectType":"ExecutionGroup"}",
"@version" => "1",
"@timestamp" => "2018-07-18T01:43:30.128Z",
"path" => "/home/murugar/Documents/kvoptions.log",
"host" => "0.0.0.0",
"type" => "file",
"msg" => "{"Name":"admin","ConfigurationObjectType":"ExecutionGroup"}",
""Name"" => "admin",
""ConfigurationObjectType"" => "ExecutionGroup"
}

Expected Ouput:

{
{
"message" => "{"Name":"admin","ConfigurationObjectType":"ExecutionGroup"}",
"@version" => "1",
"@timestamp" => "2018-07-18T01:43:30.128Z",
"path" => "/home/murugar/Documents/kvoptions.log",
"host" => "0.0.0.0",
"type" => "file",
"msg" => "{"Name":"admin","ConfigurationObjectType":"ExecutionGroup"}",
""Name"" => "admin",

}

}

In other word, the last bolded line should not appear. Why isnt exclude_keys working?

2 
grok { match => ["message","%{GREEDYDATA:msg}"] }

What are you trying to achieve here? It just copies one field to another. Why not parse message instead of msg?

Also, your input is valid json. Why not parse it using

json { source => "message" }

You could then remove fields using

mutate { remove_field => [ "ConfigurationObjectType" ] }

To actually answer you question... your field names include double quotes. If you really want to use kv then set

exclude_keys => ['"ConfigurationObjectType"']
3

Yes. It worked. I did try with "" but i missed '....'. ( ["ConfigurationObjectType"] didnt work)

In other words , do all the fields need to represented within ['anyfiledname'] (are square brackets and ' necessary?)

4

If you use kv then your field names will contain quotes, so yes, you need to include the quotes in the field names.

As I said, if you use a json filter instead of a kv filter then the field names will not contain quotes and your life will be much easier.

5

sure.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.