Kv filter for solving key value pair type of situation and avoiding some values in between

[Shivom_Kumar]

I have my log something like this :
May 23 2018 06:32:47 GMT: INFO (xdr): (xdr.c:607) [030]: dc-state CLUSTER_UP timelag-sec 0 lst 1527057166864 mlst 1527057166864 (2018-05-23 06:32:46 GMT) fnlst 0 (-) wslst 0 (-) shlat-ms 1 rsas-ms 0.000 rsas-pct 0.0 con 384 errcl 502 errsrv 1182 sz 6

i parsed it using grok filter and i have something called list_field value.
list_field_value = dc-state CLUSTER_UP timelag-sec 0 lst 1527057166864 mlst 1527057166864 (2018-05-23 06:32:46 GMT) fnlst 0 (-) wslst 0 (-) shlat-ms 1 rsas-ms 0.000 rsas-pct 0.0 con 384 errcl 502 errsrv 1182 sz 6
it is a combination of field and value.
here dc-state is a field and CLUSTER-UP is its respective value and so on.
now i am using KV filter in logstash to split list_field_value into a key value pair.
now i want to ignore two types of values while spiltting : one is (2018-05-23 06:32:46 GMT) like timstasmp and other is (-).
my code is :
grok {

		patterns_dir => ["/etc/logstash/patterns"]
		match => { "message" => "%{TIMESTAMP_LOG:time}: %{LOGLEVEL:loglevel} \(%{WORD:module}\): \(%{FILENAME:file_name}:%{NUMBER:file_no}\) ((\{%{NAMESPACE:namespace}\}|\s*%{CATEGORY:category}:|\s*%{CUSTOM_VALUE_LIST_COMPLEX:list_field_value}|\s*%{FIELD:field}\s*)+)"} 
	}

	kv{
		source => "list_field_value"
		target => "kvpairs"
		value_split => " "
    }        

i tried splitting on the basis of " " but then these two fields are also coming and timestamp is breaking and getting split and some part of it is coming as key and some part in value. because of this the other key value pair are getting affected

output is :

{
"module" => "xdr",
"@version" => "1",
"file_num" => "607",
"kvpairs" => {
"mlst" => "1527057166864",
"0.000" => "rsas-pct",
"502" => "errsrv",
"timelag-sec" => "0",
"(2018-05-23" => "06:32:46",
"1182" => "sz",
"dc-state" => "CLUSTER_UP",
"GMT)" => "fnlst",
"1" => "rsas-ms",
"wslst" => "0",
"0" => "-",
"384" => "errcl",
"(-)" => "shlat-ms",
"0.0" => "con",
"lst" => "1527057166864"
},
"path" => "/home/shivi/Desktop/c.log",
"file_name" => "xdr.c",
"time" => "May 23 2018 06:32:47 GMT",
"before" => "May 23 2018 06:32:47 GMT: INFO (xdr):",
"list_field_value" => "dc-state CLUSTER_UP timelag-sec 0 lst 1527057166864 mlst 1527057166864 (2018-05-23 06:32:46 GMT) fnlst 0 (-) wslst 0 (-) shlat-ms 1 rsas-ms 0.000 rsas-pct 0.0 con 384 errcl 502 errsrv 1182 sz 6",
"host" => "shivi-Inspiron-5558",
"message" => "May 23 2018 06:32:47 GMT: INFO (xdr): (xdr.c:607) [030]: dc-state CLUSTER_UP timelag-sec 0 lst 1527057166864 mlst 1527057166864 (2018-05-23 06:32:46 GMT) fnlst 0 (-) wslst 0 (-) shlat-ms 1 rsas-ms 0.000 rsas-pct 0.0 con 384 errcl 502 errsrv 1182 sz 6",
"@timestamp" => 2018-06-06T07:06:11.223Z,
"filename" => "xdr.c",
"loglevel" => "INFO",
"category" => "[030]",
"file_no" => "607"
}

can somone plese tell me how to ignore these certain values and keys in kvpairs ?
any help is appreciated.

[Shivom_Kumar]

i came somewhat close .
this is my new kv config :
grok {

		patterns_dir => ["/etc/logstash/patterns"]
		match => { "message" => "%{TIMESTAMP_LOG:time}: %{LOGLEVEL:loglevel} \(%{WORD:module}\): \(%{FILENAME:file_name}:%{NUMBER:file_no}\) ((\{%{NAMESPACE:namespace}\}|\s*%{CATEGORY:category}:|\s*%{CUSTOM_VALUE_LIST_COMPLEX:message}\s*|\s*%{FIELD:field}\s*)+)"}

		overwrite => ["message"]


	}


	grok {
		patterns_dir => ["/etc/logstash/patterns"]
		match => {"message" => "%{GREEDYDATA:useful_data}"}
	}  

	kv {

		source => "useful_data"
		target => "kvpairs"
		trim_key => " "
		trim_value => " "
		field_split_pattern => "\s*\(.*\)\s*|\s"
		#field_split => " "
		value_split => "\s"
	}

output is :

{
"kvpairs" => {
"lst" => "1527057166864",
"dc-state" => "CLUSTER_UP",
"mlst" => "1527057166864",
"timelag-sec" => "0"
},
"file_num" => "607",
"path" => "/home/shivi/Desktop/c.log",
"category" => "[030]",
"before" => "May 23 2018 06:32:47 GMT: INFO (xdr):",
"time" => "May 23 2018 06:32:47 GMT",
"@timestamp" => 2018-06-06T09:30:13.318Z,
"host" => "shivi-Inspiron-5558",
"message" => "dc-state CLUSTER_UP timelag-sec 0 lst 1527057166864 mlst 1527057166864 (2018-05-23 06:32:46 GMT) fnlst 0 (-) wslst 0 (-) ",
"@version" => "1",
"filename" => "xdr.c",
"loglevel" => "INFO",
"module" => "xdr",
"file_name" => "xdr.c",
"file_no" => "607",
"useful_data" => "dc-state CLUSTER_UP timelag-sec 0 lst 1527057166864 mlst 1527057166864 (2018-05-23 06:32:46 GMT) fnlst 0 (-) wslst 0 (-) "
}

as you can see it is not taking that whole timestamp thing as key but it is ignoring the fnlst as key and its respective value 0 as key-value pair .
can someone tell what is going wrong ?

[Shivom_Kumar]

ok i got somewhat more close . i was able to parse my line .
my new config file is :
kv {

		source => "useful_data"
		target => "kvpairs"
		trim_key => " "
		trim_value => " "
		field_split_pattern => "\s*\(-\)\s*|\s*\(.*[G][M][T]\)\s*|\s"
		value_split => "\s"
	}

output i got is :

    "kvpairs" => {
          "errcl" => "502",
       "dc-state" => "CLUSTER_UP",
            "lst" => "1527057166864",
          "wslst" => "0",
        "rsas-ms" => "0.000",
       "shlat-ms" => "1",
             "sz" => "6",
          "fnlst" => "0",
            "con" => "384",
           "mlst" => "1527057166864",
         "errsrv" => "1182",
       "rsas-pct" => "0.0",
    "timelag-sec" => "0"
}

this is the required output in this case.
but there is one problem. when i am taking log soething like this :

May 23 2018 06:31:47 GMT: INFO (xdr): (xdr_dlog.c:91) dlog: free-pct 100 reclaimed 57100 glst 1527057106443 (2018-05-23 06:31:46 GMT)

the output i am getting is this :

    "kvpairs" => {
       "free-pct" => "100",
      "reclaimed" => "57100",
           "glst" => "1527057106443",
    "(2018-05-23" => "06:31:46"
}

the timestamp is coming in key-value pair. i am not able to figure out what the problem is.