# grok/KV filter for multiple values

**URL:** <https://discuss.elastic.co/t/grok-kv-filter-for-multiple-values/206109>\
**Category:** Logstash\
**Created:** [October 31, 2019, 11:41pm UTC](https://discuss.elastic.co/t/grok-kv-filter-for-multiple-values/206109 "2019-10-31T23:41:04Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Senthil\_ak](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@Senthil\_ak](https://discuss.elastic.co/u/Senthil_ak)\
**Post date:** [October 31, 2019, 11:41pm UTC](https://discuss.elastic.co/t/grok-kv-filter-for-multiple-values/206109/1 "2019-10-31T23:41:04Z")

</div>

For the nginx extended logs, I am using below grok pattern and KV pattern. But the value sometimes comes as '-' , some time real values and hypen + value.

```
%{GREEDYDATA:[nginx][access][message]}'

Log - 3 types
rt=0.006 uct="-, 0.000" uht="-, 0.006" urt="0.001, 0.006" csi="-"
rt=0.016 uct="-" uht="-" urt="-" csi="-" ua="-" us="-"
rt=0.003 uct="0.000" uht="0.001" urt="0.001" csi="-" ua="-" us="304"

  kv {
    source => ["[nginx][access][message]" ]
    remove_field => ["[nginx][access][message]" ]
    field_split => " "
    value_split => "="
    trim_key => " "
    trim_value => " "
    target => "kv_temp"
  }
  if [kv_temp] {
    mutate {
      merge => { "[nginx][access]" => "kv_temp" }
      remove_field => "kv_temp"
    }
  }

```

How do i drop any field which has hypen ?  
How do i sepearate the value if that contains - and values by dropping hypen ?  
How i convert them to Integer or Number ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 1, 2019, 2:21pm UTC](https://discuss.elastic.co/t/grok-kv-filter-for-multiple-values/206109/2 "2019-11-01T14:21:32Z")

</div>

If you use trim\_value =\> "-, " it will remove the hyphens. You can use mutate+split to get an array from urt="0.001, 0.006". You can use mutate+convert to convert the fields to integer or float. To remove the empty fields you can use ruby

```
    ruby {
        code => '
            event.get("kv_temp").each { |k, v|
                if v == ""
                    event.remove("[kv_temp][#{k}]")
                end
            }
        '
    }
```

---

<div class="post-metadata">

**Author:** ![Senthil\_ak](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@Senthil\_ak](https://discuss.elastic.co/u/Senthil_ak)\
**Post date:** [November 1, 2019, 3:05pm UTC](https://discuss.elastic.co/t/grok-kv-filter-for-multiple-values/206109/3 "2019-11-01T15:05:24Z")

</div>

Thanks Much for the suggestion, I ended up using two pattern and split rather than KV split. like

`us="(%{INT:[nginx][access][upstream][status]:int}|-)"`

and if grok failure

`us="(%{DATA:[nginx][access][upstream][status]}|-)"`

Then a split for to separate values

`split => ["[nginx][access][upstream][status]", "," ]`

Then a convert to convert it to integer inside the if loop of grok failure.

`"[nginx][access][upstream][status]" => "integer"`

Now all working fine but could you please modify the ruby code check in all fields and if that any thing matches to hyphen - then remove that field ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 1, 2019, 3:37pm UTC](https://discuss.elastic.co/t/grok-kv-filter-for-multiple-values/206109/4 "2019-11-01T15:37:43Z")

</div>

If you make your grok pattern more specific than DATA then it simply will not capture - and you will not have to worry about removing it.

---

<div class="post-metadata">

**Author:** ![Senthil\_ak](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@Senthil\_ak](https://discuss.elastic.co/u/Senthil_ak)\
**Post date:** [November 1, 2019, 4:13pm UTC](https://discuss.elastic.co/t/grok-kv-filter-for-multiple-values/206109/5 "2019-11-01T16:13:18Z")

</div>

How to apply grok after split or on split value ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 1, 2019, 4:15pm UTC](https://discuss.elastic.co/t/grok-kv-filter-for-multiple-values/206109/6 "2019-11-01T16:15:19Z")

</div>

What does your data look like and what does your configuration look like?

---

<div class="post-metadata">

**Author:** ![Senthil\_ak](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@Senthil\_ak](https://discuss.elastic.co/u/Senthil_ak)\
**Post date:** [November 1, 2019, 4:17pm UTC](https://discuss.elastic.co/t/grok-kv-filter-for-multiple-values/206109/7 "2019-11-01T16:17:15Z")

</div>

> [@Senthil\_ak](#):
>
> Log - 3 types rt=0.006 uct="-, 0.000" uht="-, 0.006" urt="0.001, 0.006" csi="-" rt=0.016 uct="-" uht="-" urt="-" csi="-" ua="-" us="-" rt=0.003 uct="0.000" uht="0.001" urt="0.001" csi="-" ua="-" us="304"

Data look like above

and grok pattern for that is

`us="(%{INT:[nginx][access][upstream][status]:int}|-)"`

or

`us="(%{DATA:[nginx][access][upstream][status]}|-)"`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 29, 2019, 4:20pm UTC](https://discuss.elastic.co/t/grok-kv-filter-for-multiple-values/206109/8 "2019-11-29T16:20:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.