# Grok filter server logs with kv plugin

**URL:** <https://discuss.elastic.co/t/grok-filter-server-logs-with-kv-plugin/131336>\
**Category:** Logstash\
**Created:** [May 10, 2018, 5:25pm UTC](https://discuss.elastic.co/t/grok-filter-server-logs-with-kv-plugin/131336 "2018-05-10T17:25:01Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ram\_kriz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_kriz/32/42891_2.png) [@ram\_kriz](https://discuss.elastic.co/u/ram_kriz)\
**Post date:** [May 10, 2018, 5:25pm UTC](https://discuss.elastic.co/t/grok-filter-server-logs-with-kv-plugin/131336/1 "2018-05-10T17:25:02Z")

</div>

Hi All,  
I have started using the ELK stack(6.2) version and trying to parse the below logs.

Input:  
018-04-20T21:51:06.365164500-0400 level=7 [Proc1] [Proc2] [ID=22] [0001232800011fe8] [01285 01388] [Proc3] "[key1:N/A key2:0 key3:N/A APN:N/A key4:N/A State:N/A key5:N/A key6:0xN/A key7:N/A]: processing data

I used below filter and it matches the data.

%{TIMESTAMP\_ISO8601:logTimestamp} level=%{NUMBER:level} [%{WORD:primary\_logcomponent}] [%{WORD:secondary\_logcomponent}] [ID=%{BASE10NUM:logId}] [%{WORD:taskId1}] [%{DATA:taskId2}] [%{DATA:taskName}] "[%{GREEDYDATA:logString}

Issue:  
I can have number key values pair followed by some used defined text. I need to retrieve all the key values. I guess I need to use the KV filter(with : as delimiter) to achieve this? Also I need to ignore the user defined string at the end.

---

<div class="post-metadata">

**Author:** ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)\
**Post date:** [May 10, 2018, 5:41pm UTC](https://discuss.elastic.co/t/grok-filter-server-logs-with-kv-plugin/131336/2 "2018-05-10T17:41:52Z")

</div>

Hi,

I tried you pattern on the data provided but it failed ([you can test your grok filters here](http://grokdebug.herokuapp.com/)). I changed the pattern and now it works (for me)

`%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} \[%{WORD:primary_logcomponent}\] \[%{WORD:secondary_logcomponent}\] \[ID=%{BASE10NUM:logId}\] \[%{WORD:taskId1}\] \[%{DATA:taskId2}\] \[%{DATA:taskName}\] \"\[%{GREEDYDATA:logString}\]: %{WORD}`

If I understand you correctly you need the values from logString right? If so, you can first grok you data and run the logString trough kv filter. The config could look something like this (untested):

```
filter {
  grok {
    match => {"message" => "%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} \[%{WORD:primary_logcomponent}\] \[%{WORD:secondary_logcomponent}\] \[ID=%{BASE10NUM:logId}\] \[%{WORD:taskId1}\] \[%{DATA:taskId2}\] \[%{DATA:taskName}\] \"\[%{GREEDYDATA:logString}\]: %{WORD}"}
  }
  kv {
    source => "logString"
  }
}

```

Hope this helps,  
Paul.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 10, 2018, 5:47pm UTC](https://discuss.elastic.co/t/grok-filter-server-logs-with-kv-plugin/131336/3 "2018-05-10T17:47:55Z")

</div>

> [@ram\_kriz](#):
>
> I guess I need to use the KV filter(with : as delimiter) to achieve this?

Exactly. This will parse it...

```auto
kv { source => "logString" value_split => ":" }

```

---

<div class="post-metadata">

**Author:** ![ram\_kriz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_kriz/32/42891_2.png) [@ram\_kriz](https://discuss.elastic.co/u/ram_kriz)\
**Post date:** [May 15, 2018, 4:26pm UTC](https://discuss.elastic.co/t/grok-filter-server-logs-with-kv-plugin/131336/4 "2018-05-15T16:26:08Z")

</div>

Thanks, it works perfect. One more query as my server log highly unstructured, I need to add some more checks.  
E.g In my example primary\_logcompenent is PROC1 then I need to match a specific grok pattern.If it is PROC3 I wil apply some other filter. I am trying to figure out the right syntax to include this check . can you give me some pointers?

filter {  
grok {  
match =\> {"message" =\> "%{TIMESTAMP\_ISO8601:logTimestamp} level=%{NUMBER:level} [%{WORD:primary\_logcomponent}][%{GREEDYDATA:logString}]  
}  
//if check here if log

}

---

<div class="post-metadata">

**Author:** ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)\
**Post date:** [May 15, 2018, 4:55pm UTC](https://discuss.elastic.co/t/grok-filter-server-logs-with-kv-plugin/131336/5 "2018-05-15T16:55:43Z")

</div>

```
filter {
  grok {
       match => {"message" => "%{TIMESTAMP_ISO8601:logTimestamp} level=%{NUMBER:level} [%{WORD:primary_logcomponent}][%{GREEDYDATA:logString}]
  }
  if [primary_logcomponent] == 'PROC1' {
     // do new grok
  } else if [primary_logcomponent] == 'PROC2' {
    // do other grok
  }
}

```

I think you get the idea... 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 12, 2018, 4:55pm UTC](https://discuss.elastic.co/t/grok-filter-server-logs-with-kv-plugin/131336/6 "2018-06-12T16:55:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.