How to parse multipart json in logstash

[kunal_p]

We are having sample json logs as below

{
"responseBody": {
"Info": [ {
"supplierBookReferences": [ {
"field1": "NA",
"field2": "NA",
"field3": "",
"field4": "NA",
"field5": "",
"field6": ""
}],
"status": "Confirmed"
}],
"field1": "NA"
},
"responseHeader": {
"clientContext": {
"field7": "NA",
"field8": "NA"
},
"field9": "NA",
"field10": "abc",
"field11": "abc"
}
}

I am unable to parse the logs with the below logstash configuration.

input
{
file
{
path => "/elk/logstash-7.0.0/bin/multistepJSON.txt"
start_position => "beginning"

            }

}
filter
{
grok
{
match => [ "message", "%{GREEDYDATA:json_payload}"]
}
mutate
{
gsub => ["json_payload","[\r]",""]
}
json
{
source => "json_payload"
#target => "payload"
}
mutate
{
gsub =>["json_payload","\n\t",""]
}

}
output
{
stdout { codec => rubydebug}
}

With this configuration getting jsonfarsefailure

Any help would be appreciated.

[Badger]

grok { match => [ "message", "%{GREEDYDATA:json_payload}"] }

It does not make sense to use grok to copy the whole of one field to another. Use mutate+copy instead

mutate { gsub => ["json_payload","[\r]",""] }
mutate { gsub =>["json_payload","\n\t",""] }

Unless you have config.support_escapes enabled these do not do what you think they do. If you need to get rid of \n in a string then use a literal newline in the pattern

mutate { gsub =>["json_payload","
",""] }

That said, the json filter will work around newline, carriage return and tab, so you probably do not need to remove them.

A file input will consume a file one line at a time, so unless your JSON is a single line you will need to use a multiline codec to ingest it.

[kunal_p]

Thank you , your idea of multine worked , will keep you posted fori further ..