Grok filter server logs with kv plugin

Elastic Stack Logstash
1

Hi All,
I have started using the ELK stack(6.2) version and trying to parse the below logs.

Input:
018-04-20T21:51:06.365164500-0400 level=7 [Proc1] [Proc2] [ID=22] [0001232800011fe8] [01285 01388] [Proc3] "[key1:N/A key2:0 key3:N/A APN:N/A key4:N/A State:N/A key5:N/A key6:0xN/A key7:N/A]: processing data

I used below filter and it matches the data.

%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} [%{WORD:primary_logcomponent}] [%{WORD:secondary_logcomponent}] [ID=%{BASE10NUM:logId}] [%{WORD:taskId1}] [%{DATA:taskId2}] [%{DATA:taskName}] "[%{GREEDYDATA:logString}

Issue:
I can have number key values pair followed by some used defined text. I need to retrieve all the key values. I guess I need to use the KV filter(with : as delimiter) to achieve this? Also I need to ignore the user defined string at the end.

2

Hi,

I tried you pattern on the data provided but it failed (you can test your grok filters here). I changed the pattern and now it works (for me)

%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} \[%{WORD:primary_logcomponent}\] \[%{WORD:secondary_logcomponent}\] \[ID=%{BASE10NUM:logId}\] \[%{WORD:taskId1}\] \[%{DATA:taskId2}\] \[%{DATA:taskName}\] \"\[%{GREEDYDATA:logString}\]: %{WORD}

If I understand you correctly you need the values from logString right? If so, you can first grok you data and run the logString trough kv filter. The config could look something like this (untested):

filter {
  grok {
    match => {"message" => "%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} \[%{WORD:primary_logcomponent}\] \[%{WORD:secondary_logcomponent}\] \[ID=%{BASE10NUM:logId}\] \[%{WORD:taskId1}\] \[%{DATA:taskId2}\] \[%{DATA:taskName}\] \"\[%{GREEDYDATA:logString}\]: %{WORD}"}
  }
  kv {
    source => "logString"
  }
}

Hope this helps,
Paul.

3

Exactly. This will parse it...

kv { source => "logString" value_split => ":" }
4

Thanks, it works perfect. One more query as my server log highly unstructured, I need to add some more checks.
E.g In my example primary_logcompenent is PROC1 then I need to match a specific grok pattern.If it is PROC3 I wil apply some other filter. I am trying to figure out the right syntax to include this check . can you give me some pointers?

filter {
grok {
match => {"message" => "%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} [%{WORD:primary_logcomponent}][%{GREEDYDATA:logString}]
}
//if check here if log

}

5 
filter {
  grok {
       match => {"message" => "%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} [%{WORD:primary_logcomponent}][%{GREEDYDATA:logString}]
  }
  if [primary_logcomponent] == 'PROC1' {
     // do new grok
  } else if [primary_logcomponent] == 'PROC2' {
    // do other grok
  }
}

I think you get the idea... :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.