How to parse the multiline and nested json file

[ycui56 (yc)]

Hi,

I read lots of posts on the similar topics, but I still have problems to figure it out. So I have to post this one and hopefully get help from here. Thanks a lot in advance.

Jason file:
{
"lable1":
{
d1: 0,
d2: 0,
d3: 0,
d4: 0
},
"lable2":
{
d2: 0,
d4: 0
},
"lable3":
{
d1: 0,
d3: 0
}
}

Expect Output in Kibana:
3 separate Documents:
For example:
dname: label1
d1: 0
d2: 0
d3: 0
d4: 0

My current logstash.conf:

input {
        tcp {
                port => 9400
        }

        file {
            type => "json"
            path => "./temp/*.json"
            codec => multiline {
            pattern => "^\{|\}"
            negate => true
            what => "previous"
            auto_flush_interval => 1
            #multiline_tag => ""
            }
            start_position => "beginning"
            sincedb_path => "/dev/null"
        }
}

filter {
      #mutate { gsub => [ "message", "\A", "{", "message", "\Z", "}" ] }
      json { source => "message"}
}

[Badger]

Instead of trying to use the multiline filter to pick out each object and then mutate it into valid JSON, I would take the whole file as a single event and then restructure the result. For the input I would use

    file {
        path => "/home/ec2-user/t.test/foo.txt"
        codec => multiline {
            pattern => "^Spalanzani"
            negate => true
            what => "previous"
            auto_flush_interval => 1
            multiline_tag => ""
        }
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }

Note that file paths must be absolute. You cannot use "./temp/*.json"

Then restructure it using

    json { source => "message" target => "[@metadata][json]" remove_field => [ "message" ] }
    ruby {
        code => '
            json = event.remove("[@metadata][json]")
            if json.is_a? Hash
                newJson = []
                json.each { |k, v|
                    newJson << v.merge({ "dname" => k })
                }
                event.set("[@metadata][dname]", newJson)
            end
        '
    }
    split { field => "[@metadata][dname]" }
    ruby {
        code => '
            d = event.remove("[@metadata][dname]")
            if d.is_a? Hash
                d.each { |k, v|
                    event.set(k, v)
                }
            end
        '
    }