How to properly parse multi line json?

Elastic Stack Logstash
1

Hello all,

I am having trouble parsing the following json input in tpc

{
	"color": {
		"pixel": "152",
		"shade": "grey"
	},
	"color": {
		"pixel": "157",
		"shade": "dark"
	}
}

Result I want : 2 documents into Elastic, 1 document for each color with the field pixel and shade

Things I've attempted

Input tcp json codec : Doesn't work, it treats the event as single line
Using multiline codec below, it doesn't work either, I am getting json failure

codec => multiline {
        pattern => "^\{$"
        negate => true
        what => previous
      }

Using plain input but a split in filter

JSON 
filter {
 json {
    source => "message"
 }
 split{
  field=> "colors"
 }
}

Any hint, help?

2

A Google search suggests that JSON parsers generally overwrite the value of a duplicate key. That is certainly the case for the parser that logstash uses. I think you will need a custom parser in a ruby filter.

3

Yes it overrides the field with duplicated key but using a split should create new events.

4

You cannot split a field that only has a single value.

5

I managed to split with with multine pattern => '"color"'

Now i am getting this in the message field

"color": {
"pixel": "152",
"shade": "grey"
}

Trying to split it so pixel becomes a key and shade becomes a key

6

If you transform that into JSON using

mutate { gsub => [ "message", "\A", "{", "message", "\Z", "}" ] }

then you can use a json filter to parse it.