# Http://stackoverflow.com/questions/41427129/add-field-logstash-after-converting-value-to-json

**URL:** <https://discuss.elastic.co/t/http-stackoverflow-com-questions-41427129-add-field-logstash-after-converting-value-to-json/70426>\
**Category:** Logstash\
**Created:** [January 3, 2017, 12:12pm UTC](https://discuss.elastic.co/t/http-stackoverflow-com-questions-41427129-add-field-logstash-after-converting-value-to-json/70426 "2017-01-03T12:12:31Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yatender\_Singh](https://avatars.discourse-cdn.com/v4/letter/y/4af34b/32.png) [@Yatender\_Singh](https://discuss.elastic.co/u/Yatender_Singh)\
**Post date:** [January 3, 2017, 12:12pm UTC](https://discuss.elastic.co/t/http-stackoverflow-com-questions-41427129-add-field-logstash-after-converting-value-to-json/70426/1 "2017-01-03T12:12:31Z")

</div>

1 - unable to fetch values inside JSON.  
2 - unable to map timestamp from data.

For all general issues, please provide the following details for fast resolution:

- Version:5.1.1
- Operating System: Mac Sierra
- Config File (if you have sensitive info, please remove it):

sample data

```auto
[debug] <0.545.0> <<"{\"user_id\":\"79\",\"timestamp\":\"2016-12-28T11:10:26Z\",\"operation\":\"ver3 - Requested for recommended,verified handle information\",\"data\":\"\",\"content_id\":\"\",\"channel_id\":\"\"}">>

```

```auto
    input {
# beats {
# port => "5043"
# }
     stdin {
        }
}
    filter{
             grok {
                match => { "message" => "%{TIMESTAMP_ISO8601:syslogtimestamp} [%{SYSLOG5424SD:logtype}] <% . {BASE16FLOAT:processid}.0> <<%{QS:mydata}>>"}

        }
        json {
           source => "mydata"
           target => "parsedJson"
         add_field => {
      "user_id" => "%{[parsedJson][user_id]}}"
      "operation" => "%{[parsedJson][operation]}}"
      "data" => "%{[parsedJson][data]}}"
     }
# remove_field=>["mydata"]
        }
       mutate {
          add_field => {
               "user_id" => "%{[parsedJson][user_id]}"
               "operation" => "%{[parsedJson][operation]}"
               "data" => "%{[parsedJson:data]}"
          }
        }
}
output {
        stdout {}
    elasticsearch {
        hosts => ["localhost:9200"]
    }
}

```

- Sample Data:2016-12-28 00:16:40:53.290 [debug] \<0.545.0\> \<\<"{"user\_id":"79","timestamp":"2016-12-28T11:10:26Z","operation":"ver3 - Requested for recommended,verified handle information","data":"","content\_id":"","channel\_id":""}"\>\>

- Steps to Reproduce:

> <https://stackoverflow.com/questions/41427129/add-field-logstash-after-converting-value-to-json>

second is timestamp not matching.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 4, 2017, 7:26am UTC](https://discuss.elastic.co/t/http-stackoverflow-com-questions-41427129-add-field-logstash-after-converting-value-to-json/70426/2 "2017-01-04T07:26:37Z")

</div>

Where does the `mydata` come from and what does it look like? Make sure you format configuration and log samples as preformatted text.

Please also show a resulting event, preferably from a `stdout { codec => rubydebug }` output.

---

<div class="post-metadata">

**Author:** ![Yatender\_Singh](https://avatars.discourse-cdn.com/v4/letter/y/4af34b/32.png) [@Yatender\_Singh](https://discuss.elastic.co/u/Yatender_Singh)\
**Post date:** [January 4, 2017, 8:32am UTC](https://discuss.elastic.co/t/http-stackoverflow-com-questions-41427129-add-field-logstash-after-converting-value-to-json/70426/3 "2017-01-04T08:32:00Z")

</div>

sample data is

```auto
[debug] <0.545.0> <<"{\"user_id\":\"79\",\"timestamp\":\"2016-12-28T11:10:26Z\",\"operation\":\"ver3 - Requested for recommended,verified handle information\",\"data\":\"\",\"content_id\":\"\",\"channel_id\":\"\"}">>

```

```auto
{
   "@timestamp" => 2017-01-04T08:22:51.147Z,
         "data" => "%{[parsedJson][data]}}",
    "processid" => "0.545",
      "user_id" => "%{[parsedJson][user_id]}}",
     "@version" => "1",
         "host" => "Yatenders-iMac.local",
   "parsedJson" => "{\"user_id\":\"79\",\"timestamp\":\"2016-12-28T11:10:26Z\",\"operation\":\"ver3 - Requested for recommended,verified handle information\",\"data\":\"\",\"content_id\":\"\",\"channel_id\":\"\"}",
      "message" => "[debug] <0.545.0> <<\"{\\\"user_id\\\":\\\"79\\\",\\\"timestamp\\\":\\\"2016-12-28T11:10:26Z\\\",\\\"operation\\\":\\\"ver3 - Requested for recommended,verified handle information\\\",\\\"data\\\":\\\"\\\",\\\"content_id\\\":\\\"\\\",\\\"channel_id\\\":\\\"\\\"}\">>",
    "operation" => "%{[parsedJson][operation]}}",
       "mydata" => "\"{\\\"user_id\\\":\\\"79\\\",\\\"timestamp\\\":\\\"2016-12-28T11:10:26Z\\\",\\\"operation\\\":\\\"ver3 - Requested for recommended,verified handle information\\\",\\\"data\\\":\\\"\\\",\\\"content_id\\\":\\\"\\\",\\\"channel_id\\\":\\\"\\\"}\"",
         "tags" => []
}

```

This is rubydebug

---

<div class="post-metadata">

**Author:** ![Yatender\_Singh](https://avatars.discourse-cdn.com/v4/letter/y/4af34b/32.png) [@Yatender\_Singh](https://discuss.elastic.co/u/Yatender_Singh)\
**Post date:** [January 4, 2017, 8:38am UTC](https://discuss.elastic.co/t/http-stackoverflow-com-questions-41427129-add-field-logstash-after-converting-value-to-json/70426/4 "2017-01-04T08:38:14Z")

</div>

sample data

```auto
[debug] <0.545.0> <<"{\"user_id\":\"79\",\"timestamp\":\"2016-12-28T11:10:26Z\",\"operation\":\"ver3 - Requested for recommended,verified handle information\",\"data\":\"\",\"content_id\":\"\",\"channel_id\":\"\"}">>

```

when my configuration is

```auto
input {
# beats {
# port => "5043"
# }
     stdin {
        }
}
filter{
        grok {
           match => { "message" => "[%{SYSLOG5424SD:logtype}] <%{BASE16FLOAT:processid}.0> <<%{QS:mydata}>>"}

        }
# date{
# match => {"syslogtimestamp","ISO8601","yyyy-MM-dd HH:mm:ss:SSSZ"}
# }
        json {
           source => "mydata"
           target => "parsedJson"
         add_field => {
      "user_id_up" => "%{[parsedJson][user_id]}}"
      "operation_up" => "%{[parsedJson][operation]}}"
      "data_up" => "%{[parsedJson][data]}}"
     }
# remove_field=>["mydata"]
        }
        mutate {
           add_field => {
                "user_id" => "%{[parsedJson][user_id]}"
                "operation" => "%{[parsedJson][operation]}"
                "data" => "%{[parsedJson][data]}"
           }
         }
}
output {
        stdout {
                codec => rubydebug
        }
    elasticsearch {
        hosts => ["localhost:9200"]
    }
}

```

I am getting

```auto
{
            "data" => "%{[parsedJson][data]}",
         "message" => "[debug] <0.545.0> <<\"{\\\"user_id\\\":\\\"79\\\",\\\"timestamp\\\":\\\"2016-12-28T11:10:26Z\\\",\\\"operation\\\":\\\"ver3 - Requested for recommended,verified handle information\\\",\\\"data\\\":\\\"\\\",\\\"content_id\\\":\\\"\\\",\\\"channel_id\\\":\\\"\\\"}\">>",
          "mydata" => "\"{\\\"user_id\\\":\\\"79\\\",\\\"timestamp\\\":\\\"2016-12-28T11:10:26Z\\\",\\\"operation\\\":\\\"ver3 - Requested for recommended,verified handle information\\\",\\\"data\\\":\\\"\\\",\\\"content_id\\\":\\\"\\\",\\\"channel_id\\\":\\\"\\\"}\"",
            "tags" => [],
      "@timestamp" => 2017-01-04T08:36:35.631Z,
       "processid" => "0.545",
      "user_id_up" => "%{[parsedJson][user_id]}}",
         "data_up" => "%{[parsedJson][data]}}",
         "user_id" => "%{[parsedJson][user_id]}",
        "@version" => "1",
            "host" => "Yatenders-iMac.local",
      "parsedJson" => "{\"user_id\":\"79\",\"timestamp\":\"2016-12-28T11:10:26Z\",\"operation\":\"ver3 - Requested for recommended,verified handle information\",\"data\":\"\",\"content_id\":\"\",\"channel_id\":\"\"}",
    "operation_up" => "%{[parsedJson][operation]}}",
       "operation" => "%{[parsedJson][operation]}"
}

```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 4, 2017, 8:55am UTC](https://discuss.elastic.co/t/http-stackoverflow-com-questions-41427129-add-field-logstash-after-converting-value-to-json/70426/5 "2017-01-04T08:55:43Z")

</div>

Okay. `mydata` doesn't contain a JSON object but a double-quoted string that contains a JSON object. However, as it happens a double-quoted string can also be decoded as JSON so the result of your first json filter is a string with a JSON object (as you can see in your `parsedJson` field). If you send _that_ string to another json filter you'll be okay.

---

<div class="post-metadata">

**Author:** ![Yatender\_Singh](https://avatars.discourse-cdn.com/v4/letter/y/4af34b/32.png) [@Yatender\_Singh](https://discuss.elastic.co/u/Yatender_Singh)\
**Post date:** [January 4, 2017, 10:26am UTC](https://discuss.elastic.co/t/http-stackoverflow-com-questions-41427129-add-field-logstash-after-converting-value-to-json/70426/6 "2017-01-04T10:26:55Z")

</div>

Thanks a lot working fine now.

now I know use of rubydebug.

Thanks again

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 1, 2017, 10:26am UTC](https://discuss.elastic.co/t/http-stackoverflow-com-questions-41427129-add-field-logstash-after-converting-value-to-json/70426/7 "2017-02-01T10:26:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.