# KV Filtering data between square brackets

**URL:** <https://discuss.elastic.co/t/kv-filtering-data-between-square-brackets/126124>\
**Category:** Logstash\
**Created:** [March 29, 2018, 4:46pm UTC](https://discuss.elastic.co/t/kv-filtering-data-between-square-brackets/126124 "2018-03-29T16:46:34Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![harinandan04](https://avatars.discourse-cdn.com/v4/letter/h/45deac/32.png) [@harinandan04](https://discuss.elastic.co/u/harinandan04)\
**Post date:** [March 29, 2018, 4:46pm UTC](https://discuss.elastic.co/t/kv-filtering-data-between-square-brackets/126124/1 "2018-03-29T16:46:34Z")

</div>

Hi,

How can i parse data which has two layers of square brackets.

1. Inner square brackets
2. Outer square brackets

I only want to consider outer square brackets data as key value pairs.

Example : [filename=0\_578cc[R2][]2 veckor f=f600re (2).doc]

In the above example i want to take filename as key but it is also considering the inner square bracket which is breaking my logic.  
I am using grok and kv filter with the following config.

match =\> ["message", "%{SYSLOGTIMESTAMP:eventtime}\s(?[^\s]_)\s(?[^\s]_)\s(?[^\s]\*)\s%{GREEDYDATA:msg}"]

kv {  
source =\> "msg"  
field\_split =\> "]["  
value\_split =\> "="  
}

Can anyone suggest me what needs to be done.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [March 29, 2018, 5:46pm UTC](https://discuss.elastic.co/t/kv-filtering-data-between-square-brackets/126124/2 "2018-03-29T17:46:13Z")

</div>

Can you provide more examples inputs, and explicitly what you hope to extract from each? It's really hard to come up with a generic pattern that will work without clearly knowing what you want.

---

<div class="post-metadata">

**Author:** ![harinandan04](https://avatars.discourse-cdn.com/v4/letter/h/45deac/32.png) [@harinandan04](https://discuss.elastic.co/u/harinandan04)\
**Post date:** [March 29, 2018, 6:10pm UTC](https://discuss.elastic.co/t/kv-filtering-data-between-square-brackets/126124/3 "2018-03-29T18:10:56Z")

</div>

Hi, Thanks for your reply. Below is one of the classic example.

Oct 3 20:13:48 WIN-C6MQ3RMMBL IndexerI103 [event-type=indexation][guid=D2B636B35A54433AB2916FEA4D180538][filename=0\_1d1ea[\*\*R4][\*\*]2011-05-25 Rumsf=f600rdelning Byggnad\_Plan\_Rum.xls][fileext=xls][source=Internal][size=118784][converter=\_FilenameToHtmlNoBlob][success=True][message=ok][durationExecution=328][durationConversion=0][durationExts=0][durationLemma=16][durationIndexPacket=0][durationCache=16]

From this input if you see the key 'filename' has value which includes "][". This is breaking my logic of getting kvpairs using "][" as field split.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [March 29, 2018, 6:47pm UTC](https://discuss.elastic.co/t/kv-filtering-data-between-square-brackets/126124/4 "2018-03-29T18:47:27Z")

</div>

> [@harinandan04](#):
>
> Oct 3 20:13:48 WIN-C6MQ3RMMBL IndexerI103 [event-type=indexation][guid=D2B636B35A54433AB2916FEA4D180538][filename=0\_1d1ea[**R4][**]2011-05-25 Rumsf=f600rdelning Byggnad\_Plan\_Rum.xls][fileext=xls][source=Internal][size=118784][converter=\_FilenameToHtmlNoBlob][success=True][message=ok][durationExecution=328][durationConversion=0][durationExts=0][durationLemma=16][durationIndexPacket=0][durationCache=16]

I cannot tell what you expect to extract from this data; can you provide a mapping of what keys you expect to extract, and what you expect the values to be, _exactly_?

* * *

There was recently a new release of the kv filter plugin, which allows us to specify a _pattern_ for the field-splitter and value-splitter; the following may work, but it will not be especially performant because it will need to do a lot of backtracking in order to capture the right bits:

```auto
bin/logstash-plugin update logstash-filter-kv

```

Once you have done so, we can define the pattern to split fields on one of the following:

- the start of a string followed by an open-square-bracket `^\[` (cheap); OR
- a close-square-bracket followed by the end-of-line `\]$` (cheap); OR
- a close-suare-bracket and open-square-bracket that is followed by something that looks like a key `\]\[(?=[A-Za-z0-9]+=))` (expensive; may need to backtrack)

Put it together, and we get:

```auto
filter {
  kv {
    field_split_pattern => "(?:^\[|\]$|\]\[(?=[A-Za-z0-9]+=))"
  }
}

```

With the above pattern, I get:

```auto
{
                 "source" => "Internal",
                   "host" => "castrovel.local",
                "success" => "True",
     "durationConversion" => "0",
                   "size" => "118784",
      "durationExecution" => "328",
              "converter" => "_FilenameToHtmlNoBlob",
             "@timestamp" => 2018-03-29T18:43:20.270Z,
                   "guid" => "D2B636B35A54433AB2916FEA4D180538",
                "fileext" => "xls",
           "durationExts" => "0",
          "durationLemma" => "16",
             "event-type" => "indexation",
    "durationIndexPacket" => "0",
               "@version" => "1",
                "message" => "ok",
          "durationCache" => "16",
               "filename" => "0_1d1ea[**R4][**]2011-05-25 Rumsf=f600rdelning Byggnad_Plan_Rum.xls"
}

```

---

<div class="post-metadata">

**Author:** ![harinandan04](https://avatars.discourse-cdn.com/v4/letter/h/45deac/32.png) [@harinandan04](https://discuss.elastic.co/u/harinandan04)\
**Post date:** [April 2, 2018, 11:06am UTC](https://discuss.elastic.co/t/kv-filtering-data-between-square-brackets/126124/5 "2018-04-02T11:06:44Z")

</div>

Hi,

Thanks. This is working for some of the log, although using this is not getting me "user-id" in the below log.

Sep 2 14:32:57 WIN-5KCJEHGCVCM MyApp2 [event-type=search.text][guid=194B71D7E84A4AF6B9C21CBFB60E9851][user-id=ad|S-1-5-21-1292428093-776561741-11674531-9345][profile=rdSearch][session-id=E908D509BEF44A91AA1489ACC5C49461][duration=594][result-id=B4D9BDFD3D744E46A1BC75729788F51D][result-count=78043][text=test]

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 2, 2018, 4:45pm UTC](https://discuss.elastic.co/t/kv-filtering-data-between-square-brackets/126124/6 "2018-04-02T16:45:18Z")

</div>

user-id is only key that contains a hyphen. Change

```auto
field_split_pattern => "(?:^\[|\]$|\]\[(?=[A-Za-z0-9]+=))"

```

to

```auto
field_split_pattern => "(?:^\[|\]$|\]\[(?=[-A-Za-z0-9]+=))"

```

---

<div class="post-metadata">

**Author:** ![harinandan04](https://avatars.discourse-cdn.com/v4/letter/h/45deac/32.png) [@harinandan04](https://discuss.elastic.co/u/harinandan04)\
**Post date:** [April 3, 2018, 4:41am UTC](https://discuss.elastic.co/t/kv-filtering-data-between-square-brackets/126124/7 "2018-04-03T04:41:29Z")

</div>

@Badger, Thank you. This worked well.

@yaauie, Thank you mate.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 1, 2018, 4:41am UTC](https://discuss.elastic.co/t/kv-filtering-data-between-square-brackets/126124/8 "2018-05-01T04:41:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.