Extract json fields from message

Hello there,
After applying grok filter to my message I get one of the fields called main_message which looks like this

main_message     Processing data {"si":"-2","a":"-54.0","r":"0","version":"1","id":"C112"}

From the main_message I now want to extract the json and parse each key as a field into elastic so that I can query in kql like this
id : "C112"

I tried this but ended up getting json parsing error

          gsub => [ "main_message", "Processing data ", "" ]
          source => "main_message" target => "copy_main_message"
          source => "copy_main_message" target => "log_json"

error I got
exception=>#<LogStash::Json::ParserError: Unrecognized token 'Processing': was expecting ('true', 'false' or 'null')

If you use

output { stdout { codec => rubydebug } }

then what does the main_message field look like? Alternatively, expand an event in the Discover view of Kibana, switch to the JSON tab, and show us what the main_message field looks like.

Hello Badger,
Thanks for responding here it is

"transaction_id": "282",
    "@timestamp": "2021-04-09T18:58:14.644Z",
    "host": {
      "name": "hostname"
    "main_message": "Processing data {\"si\":\"-2\",\"a\":\"-54.0\",\"r\":\"0\",\"version\":\"1\",\"id\":\"C112\"}",
    "agent": {
      "type": "filebeat",
      "version": "7.10.0",
      "name": "hostname",
      "hostname": "hostname",
      "id": "2f8eb1d0-2011-447d-a7c0-c5531d3083eb",
      "ephemeral_id": "9b61977d-e1a4-4767-97a7-db31535c1393"

and the here is the error I have
:exception=>java.lang.ClassCastException: class org.jruby.RubyHash cannot be cast to class org.jruby.RubyIO (org.jruby.RubyHash and org.jruby.RubyIO are in unnamed module of loader 'app')}

Just delete the second json filter.

input { generator { count => 1 lines => [ '' ] } }
filter {
    mutate { add_field => { "main_message" => 'Processing data {"si":"-2","a":"-54.0","r":"0","version":"1","id":"C112"}' } }
    mutate { gsub => [ "main_message", "Processing data ", "" ] }
    json { source => "main_message" target => "copy_main_message" }
output  { stdout { codec => rubydebug { metadata => false } } }

will produce

"copy_main_message" => {
          "r" => "0",
         "si" => "-2",
    "version" => "1",
         "id" => "C112",
          "a" => "-54.0"

You can see that [copy_main_message] is a hash (class org.jruby.RubyHash) and a json filter cannot parse that.

Hey Badger,
Remember the main_message is one field which I extract from log file which means it keeps changing. That is a dynamic value not static.


That's fine I was just showing you what will happen for the specific value you gave. It will work for other values too if they are valid JSON.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.