Parsing nested json to flat structure

Elastic Stack Logstash
1

Hi Team,

I was looking to parse my nested json structure using json filter. I have looked at some of the posts and tried their solution.

My question, is it possible to make this structure completely flat,

So if my log message is

{"Common": {"LogLevel":"Info","Details":"Successful","Name":"Test","Browser":"chrome"}, "Application": {"Id":1,"Type":"Web"}}

and if I have my filter set up as

filter {
	json {
         source => "message"
         target => "parsedmessage"
    }
    json {
      source => "[parsedmessage][Application]"
      target => "[parsedmessage][Application]"
    }
json {
      source => "[parsedmessage][Common]"
      target => "[parsedmessage][Common]"
    }
}

a) I keep getting error for

Error parsing json {:source=>"[parsedmessage][Application]", :raw=>{"Id"=>1, "Type"=>"Web"}, :exception=>java.lang.ClassCastException: class org.jruby.RubyHash cannot be cast to class org.jruby.RubyIO (org.jruby.RubyHash and org.jruby.RubyIO are in unnamed module of loader 'app')}

b) I am just wondering, if I can make the output as a complete flat structure, like this

      "@version" => "1",           
             "Id" => 1,
             "Type" => "Web",        
             "Browser" => "chrome",
             "Details" => "Successful",
             "LogLevel" => "Info",
             "Name" => "Test"

and not some embedded json like this

      "@version" => "1",
    "parsedmessage" => {
        "Application" => {
              "Id" => 1,
            "Type" => "Web"
        },
             "Common" => {
             "Browser" => "chrome",
             "Details" => "Successful",
            "LogLevel" => "Info",
                "Name" => "Test"
        }

Any suggestions, much appreciated.

Thanks

2

The json filter will parse all of the nested JSON objects and create hashes from them. The json filter parses a string, which is why you get that error about casting a RubyHash to a RubyIO.

If you want to move the nested objects to the top level and you know their names you can use code like this.