Extract json fields from message

Elastic Stack Logstash
1

Hello there,
After applying grok filter to my message I get one of the fields called main_message which looks like this

main_message     Processing data {"si":"-2","a":"-54.0","r":"0","version":"1","id":"C112"}

From the main_message I now want to extract the json and parse each key as a field into elastic so that I can query in kql like this
id : "C112"

I tried this but ended up getting json parsing error

mutate
     {
          gsub => [ "main_message", "Processing data ", "" ]
     }
json
     {
          source => "main_message" target => "copy_main_message"
     }
json
     {
          source => "copy_main_message" target => "log_json"
     }

error I got
exception=>#<LogStash::Json::ParserError: Unrecognized token 'Processing': was expecting ('true', 'false' or 'null')

2

If you use

output { stdout { codec => rubydebug } }

then what does the main_message field look like? Alternatively, expand an event in the Discover view of Kibana, switch to the JSON tab, and show us what the main_message field looks like.

3

Hello Badger,
Thanks for responding here it is

"transaction_id": "282",
    "@timestamp": "2021-04-09T18:58:14.644Z",
    "host": {
      "name": "hostname"
    },
    "main_message": "Processing data {\"si\":\"-2\",\"a\":\"-54.0\",\"r\":\"0\",\"version\":\"1\",\"id\":\"C112\"}",
    "agent": {
      "type": "filebeat",
      "version": "7.10.0",
      "name": "hostname",
      "hostname": "hostname",
      "id": "2f8eb1d0-2011-447d-a7c0-c5531d3083eb",
      "ephemeral_id": "9b61977d-e1a4-4767-97a7-db31535c1393"
    },

and the here is the error I have
:exception=>java.lang.ClassCastException: class org.jruby.RubyHash cannot be cast to class org.jruby.RubyIO (org.jruby.RubyHash and org.jruby.RubyIO are in unnamed module of loader 'app')}

4

Just delete the second json filter.

input { generator { count => 1 lines => [ '' ] } }
filter {
    mutate { add_field => { "main_message" => 'Processing data {"si":"-2","a":"-54.0","r":"0","version":"1","id":"C112"}' } }
    mutate { gsub => [ "main_message", "Processing data ", "" ] }
    json { source => "main_message" target => "copy_main_message" }
}
output  { stdout { codec => rubydebug { metadata => false } } }

will produce

"copy_main_message" => {
          "r" => "0",
         "si" => "-2",
    "version" => "1",
         "id" => "C112",
          "a" => "-54.0"
},

You can see that [copy_main_message] is a hash (class org.jruby.RubyHash) and a json filter cannot parse that.

5

Hey Badger,
Remember the main_message is one field which I extract from log file which means it keeps changing. That is a dynamic value not static.

thanks

6

That's fine I was just showing you what will happen for the specific value you gave. It will work for other values too if they are valid JSON.