# Extract json fields from message

**URL:** <https://discuss.elastic.co/t/extract-json-fields-from-message/269742>\
**Category:** Logstash\
**Created:** [April 9, 2021, 7:31pm UTC](https://discuss.elastic.co/t/extract-json-fields-from-message/269742 "2021-04-09T19:31:45Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\
**Post date:** [April 9, 2021, 7:31pm UTC](https://discuss.elastic.co/t/extract-json-fields-from-message/269742/1 "2021-04-09T19:31:45Z")

</div>

Hello there,  
After applying grok filter to my message I get one of the fields called main\_message which looks like this

```auto
main_message Processing data {"si":"-2","a":"-54.0","r":"0","version":"1","id":"C112"}

```

From the main\_message I now want to extract the json and parse each key as a field into elastic so that I can query in kql like this  
`id : "C112" `

I tried this but ended up getting json parsing error

```auto
mutate
     {
          gsub => ["main_message", "Processing data ", ""]
     }
json
     {
          source => "main_message" target => "copy_main_message"
     }
json
     {
          source => "copy_main_message" target => "log_json"
     }

```

error I got  
`exception=>#<LogStash::Json::ParserError: Unrecognized token 'Processing': was expecting ('true', 'false' or 'null') `

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 9, 2021, 7:43pm UTC](https://discuss.elastic.co/t/extract-json-fields-from-message/269742/2 "2021-04-09T19:43:57Z")

</div>

If you use

```
output { stdout { codec => rubydebug } }

```

then what does the main\_message field look like? Alternatively, expand an event in the Discover view of Kibana, switch to the JSON tab, and show us what the main\_message field looks like.

---

<div class="post-metadata">

**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\
**Post date:** [April 9, 2021, 8:05pm UTC](https://discuss.elastic.co/t/extract-json-fields-from-message/269742/3 "2021-04-09T20:05:43Z")

</div>

Hello Badger,  
Thanks for responding here it is

```auto
"transaction_id": "282",
    "@timestamp": "2021-04-09T18:58:14.644Z",
    "host": {
      "name": "hostname"
    },
    "main_message": "Processing data {\"si\":\"-2\",\"a\":\"-54.0\",\"r\":\"0\",\"version\":\"1\",\"id\":\"C112\"}",
    "agent": {
      "type": "filebeat",
      "version": "7.10.0",
      "name": "hostname",
      "hostname": "hostname",
      "id": "2f8eb1d0-2011-447d-a7c0-c5531d3083eb",
      "ephemeral_id": "9b61977d-e1a4-4767-97a7-db31535c1393"
    },

```

and the here is the error I have  
`:exception=>java.lang.ClassCastException: class org.jruby.RubyHash cannot be cast to class org.jruby.RubyIO (org.jruby.RubyHash and org.jruby.RubyIO are in unnamed module of loader 'app')}`

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 9, 2021, 8:26pm UTC](https://discuss.elastic.co/t/extract-json-fields-from-message/269742/4 "2021-04-09T20:26:45Z")

</div>

> [@adityak248](#):
>
> :exception=\>java.lang.ClassCastException: class org.jruby.RubyHash cannot be cast to class org.jruby.RubyIO (org.jruby.RubyHash and org.jruby.RubyIO are in unnamed module of loader 'app')}

Just delete the second json filter.

```
input { generator { count => 1 lines => [''] } }
filter {
    mutate { add_field => { "main_message" => 'Processing data {"si":"-2","a":"-54.0","r":"0","version":"1","id":"C112"}' } }
    mutate { gsub => ["main_message", "Processing data ", ""] }
    json { source => "main_message" target => "copy_main_message" }
}
output { stdout { codec => rubydebug { metadata => false } } }

```

will produce

```
"copy_main_message" => {
          "r" => "0",
         "si" => "-2",
    "version" => "1",
         "id" => "C112",
          "a" => "-54.0"
},

```

You can see that [copy\_main\_message] is a hash (class org.jruby.RubyHash) and a json filter cannot parse that.

---

<div class="post-metadata">

**Author:** ![adityak248](https://avatars.discourse-cdn.com/v4/letter/a/7ea924/32.png) [@adityak248](https://discuss.elastic.co/u/adityak248)\
**Post date:** [April 9, 2021, 9:03pm UTC](https://discuss.elastic.co/t/extract-json-fields-from-message/269742/5 "2021-04-09T21:03:05Z")

</div>

Hey Badger,  
Remember the main\_message is one field which I extract from log file which means it keeps changing. That is a dynamic value not static.

thanks

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 9, 2021, 9:11pm UTC](https://discuss.elastic.co/t/extract-json-fields-from-message/269742/6 "2021-04-09T21:11:37Z")

</div>

That's fine I was just showing you what will happen for the specific value you gave. It will work for other values too if they are valid JSON.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 7, 2021, 9:12pm UTC](https://discuss.elastic.co/t/extract-json-fields-from-message/269742/7 "2021-05-07T21:12:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.