Grok filter server logs with kv plugin

[ram_kriz]

Hi All,
I have started using the ELK stack(6.2) version and trying to parse the below logs.

Input:
018-04-20T21:51:06.365164500-0400 level=7 [Proc1] [Proc2] [ID=22] [0001232800011fe8] [01285 01388] [Proc3] "[key1:N/A key2:0 key3:N/A APN:N/A key4:N/A State:N/A key5:N/A key6:0xN/A key7:N/A]: processing data

I used below filter and it matches the data.

%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} [%{WORD:primary_logcomponent}] [%{WORD:secondary_logcomponent}] [ID=%{BASE10NUM:logId}] [%{WORD:taskId1}] [%{DATA:taskId2}] [%{DATA:taskName}] "[%{GREEDYDATA:logString}

Issue:
I can have number key values pair followed by some used defined text. I need to retrieve all the key values. I guess I need to use the KV filter(with : as delimiter) to achieve this? Also I need to ignore the user defined string at the end.

[pjanzen (Paul Janzen)]

Hi,

I tried you pattern on the data provided but it failed (you can test your grok filters here). I changed the pattern and now it works (for me)

%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} \[%{WORD:primary_logcomponent}\] \[%{WORD:secondary_logcomponent}\] \[ID=%{BASE10NUM:logId}\] \[%{WORD:taskId1}\] \[%{DATA:taskId2}\] \[%{DATA:taskName}\] \"\[%{GREEDYDATA:logString}\]: %{WORD}

If I understand you correctly you need the values from logString right? If so, you can first grok you data and run the logString trough kv filter. The config could look something like this (untested):

filter {
  grok {
    match => {"message" => "%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} \[%{WORD:primary_logcomponent}\] \[%{WORD:secondary_logcomponent}\] \[ID=%{BASE10NUM:logId}\] \[%{WORD:taskId1}\] \[%{DATA:taskId2}\] \[%{DATA:taskName}\] \"\[%{GREEDYDATA:logString}\]: %{WORD}"}
  }
  kv {
    source => "logString"
  }
}

Hope this helps,
Paul.

[Badger]

@ram_kriz

I guess I need to use the KV filter(with : as delimiter) to achieve this?

Exactly. This will parse it...

kv { source => "logString" value_split => ":" }

[ram_kriz]

Thanks, it works perfect. One more query as my server log highly unstructured, I need to add some more checks.
E.g In my example primary_logcompenent is PROC1 then I need to match a specific grok pattern.If it is PROC3 I wil apply some other filter. I am trying to figure out the right syntax to include this check . can you give me some pointers?

filter {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} [%{WORD:primary_logcomponent}][%{GREEDYDATA:logString}]
}
//if check here if log

}

[pjanzen (Paul Janzen)]

filter {
  grok {
       match => {"message" => "%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} [%{WORD:primary_logcomponent}][%{GREEDYDATA:logString}]
  }
  if [primary_logcomponent] == 'PROC1' {
     // do new grok
  } else if [primary_logcomponent] == 'PROC2' {
    // do other grok
  }
}

I think you get the idea...