# Disapearing fields during kv parsing

**URL:** <https://discuss.elastic.co/t/disapearing-fields-during-kv-parsing/290652>\
**Category:** Elasticsearch\
**Created:** [December 1, 2021, 11:03am UTC](https://discuss.elastic.co/t/disapearing-fields-during-kv-parsing/290652 "2021-12-01T11:03:49Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ddoroshenko](https://avatars.discourse-cdn.com/v4/letter/d/b9e5f3/32.png) [@ddoroshenko](https://discuss.elastic.co/u/ddoroshenko)\
**Post date:** [December 1, 2021, 11:03am UTC](https://discuss.elastic.co/t/disapearing-fields-during-kv-parsing/290652/1 "2021-12-01T11:03:49Z")

</div>

Hi!

I'm trying to parse the following log record  
`__policy_id_tag:"product=APP1 & APP2[db_tag={ABCDEF}; mgmt=abcd; date=123456789; policy_name=foo]"; product:"APP1 & APP2"; mgmt:"abcd"; event:"Canceled"`

and when I use the filter

```auto
filter {
  grok {
    match => { "message" => "%{GREEDYDATA:[@metadata][message]}" }
  }

  kv {
    source => "[@metadata][message]"
    field_split => ";"
    value_split => ":"
    trim_key => " _"
    target => "fws"
  }

  grok {
    match => { "[fws][policy_id_tag]" => "%{GREEDYDATA:[@metadata][policy_id_tag]}" }
  }
}

```

I see the output

```auto
"@timestamp" => 2021-12-01T10:52:13.353Z,
      "@version" => "1",
      "sequence" => 0,
          "host" => "localhost.localdomain",
       "message" => "__policy_id_tag:\"product=APP1 & APP2[db_tag={ABCDEF}; mgmt=abcd; date=123456789; policy_name=foo]\"; product:\"APP1 & APP2\"; mgmt:\"abcd\"; event:\"Canceled\"",
           "fws" => {
              "product" => "APP1 & APP2",
                 "mgmt" => "abcd",
                "event" => "Canceled",
        "policy_id_tag" => "product=APP1 & APP2[db_tag={ABCDEF}; mgmt=abcd; date=123456789; policy_name=foo]"

```

It looks OK.

Then I add next `kv` section to parse `policy_id_tag` field

```auto
filter {
  grok {
    match => { "message" => "%{GREEDYDATA:[@metadata][message]}" }
  }

  kv {
    source => "[@metadata][message]"
    field_split => ";"
    value_split => ":"
    trim_key => " _"
    target => "fws"
  }

  grok {
    match => { "[fws][policy_id_tag]" => "%{GREEDYDATA:[@metadata][policy_id_tag]}" }
  }

  kv {
    source => "[@metadata][policy_id_tag]"
    field_split => "\[;"
    trim_key => " "
    remove_char_value => "\]"
    target => "fws"
  }
}

```

What I see the `event` and `policy_id_tag` fields dissapeared

```auto
"@version" => "1",
    "@timestamp" => 2021-12-01T10:54:05.935Z,
       "message" => "__policy_id_tag:\"product=APP1 & APP2[db_tag={ABCDEF}; mgmt=abcd; date=123456789; policy_name=foo]\"; product:\"APP1 & APP2\"; mgmt:\"abcd\"; event:\"Canceled\"",
          "host" => "localhost.localdomain",
           "fws" => {
               "mgmt" => "abcd",
        "policy_name" => "foo",
            "product" => "APP1 & APP2",
               "date" => "123456789",
             "db_tag" => "{ABCDEF}"
    },
      "sequence" => 0

```

What could be problem and how to solve it?

**Note:** without `target => "fws"` it is parsed well

```auto
"@version" => "1",
             "host" => "localhost.localdomain",
       "@timestamp" => 2021-12-01T11:34:15.475Z,
          "message" => "__policy_id_tag:\"product=APP1 & APP2[db_tag={ABCDEF}; mgmt=abcd; date=123456789; policy_name=foo]\"; product:\"APP1 & APP2\"; mgmt:\"abcd\"; event:\"Canceled\"",
         "sequence" => 0,
      "policy_name" => "foo",
          "product" => "APP1 & APP2",
            "event" => "Canceled",
    "policy_id_tag" => "product=APP1 & APP2[db_tag={ABCDEF}; mgmt=abcd; date=123456789; policy_name=foo]",
             "mgmt" => "abcd",
             "date" => "123456789",
           "db_tag" => "{ABCDEF}"

```

---

<div class="post-metadata">

**Author:** ![ddoroshenko](https://avatars.discourse-cdn.com/v4/letter/d/b9e5f3/32.png) [@ddoroshenko](https://discuss.elastic.co/u/ddoroshenko)\
**Post date:** [December 1, 2021, 1:20pm UTC](https://discuss.elastic.co/t/disapearing-fields-during-kv-parsing/290652/2 "2021-12-01T13:20:03Z")

</div>

Well, the problem is `kv` rewrites `fws` field.

```auto
filter {
  grok {
    match => { "message" => "%{GREEDYDATA:[@metadata][message]}" }
  }

  kv {
    source => "[@metadata][message]"
    field_split => ";"
    value_split => ":"
    trim_key => " _"
    target => "fws"
  }

  grok {
    match => { "[fws][policy_id_tag]" => "%{GREEDYDATA:[@metadata][policy_id_tag]}" }
  }

  kv {
    source => "[@metadata][policy_id_tag]"
    field_split => "\[;"
    trim_key => " "
    remove_char_value => "\]"
    target => "fws_tmp"
  }

 mutate {
    merge => { "fws" => "fws_tmp" }
  }
}

```

solved the problem

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 29, 2021, 1:20pm UTC](https://discuss.elastic.co/t/disapearing-fields-during-kv-parsing/290652/3 "2021-12-29T13:20:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.