How to parse message with random data using logstash filter?

I have received the below message from the producer at Axon Topic with some random ASCII data.
"message" => "\u0000\u0000\u0000\u0002�\u0004C1�\u0001\u0002�\u0003\u0000\u0000\u0000\u0000A�\u0003{"prdctId":"10077","id":"10077","name":"ComPass","owner":{"id":"E040547","name":"Shashi Raghunandan"},"alias":[{"id":"a1Id","name":"a1Name"},{"id":"a2Id","name":"a2Name"}],"cntryAvail":["n/a"]}�����[\u0002\u0000\u0002\u0000"

I have tried split filter on open curly braces '{' to take out the outer json data from the message.
mutate {
split => ["message", "{"]
add_field => { "splitmsg" => "%{[message][0]}" }
add_field => { "msg" => "%{[message][1]}" }
}

but then we get the splitted messages from the message field, which seems not a correct way to do it as after that we got the message in below format.
{"prdctId":"10077","id":"10077","name":"ComPass","owner":{"id":"E040547","name":"Shashi Raghunandan"},"alias":[{"id":"a1Id","name":"a1Name"},{"id":"a2Id","name":"a2Name"}],"cntryAvail":["n/a"]}

Now, I'm not able to remove backslashes from the message. Please suggest the filter to be applied in such a way to have the output as json from above message into inline format.
{"prdctId":"10077","id":"10077","name":"ComPass","owner":{"id":"E040547","name":"Shashi Raghunandan"},"alias":[{"id":"a1Id","name":"a1Name"},{"id":"a2Id","name":"a2Name"}],"cntryAvail":["n/a"]}

You could try something like

mutate { gsub => [ "message", "^[^{]+", "", "message", "}[^}]+$", "" ] }

which would remove everything before the first { and everything after the last }.

After trying above filter, got message as given below.

{
    "_index" : "test-product-logstash-22",
    "_type" : "_doc",
    "_id" : "%{id}",
    "_score" : 1.0,
    "_source" : {
      "message" : "{\"prdctId\":\"10077\",\"id\":\"10077\",\"name\":\"ComPass\",\"status\":\"In Development\",\"family\":\"Strategic Growth\",\"line\":\"Strategic Growth\",\"shortDesc\":\"Enables interoperable digital IDs\",\"longDesc\":\"A platform that enables interoperable digital ID's and other capabilities \",\"owner\":{\"id\":\"E040547\",\"name\":\"Shashi Raghunandan\"},\"program\":{\"id\":\"422\",\"name\":\"Regional Solutions and Bundles\"},\"alias\":[{\"id\":\"a1Id\",\"name\":\"a1Name\"},{\"id\":\"a2Id\",\"name\":\"a2Name\"}],\"regAvail\":[\"ap\",\"lac\",\"mea\"],\"cntryAvail\":[\"Global\"],\"salesCenterUrl\":\"salesCenterUrlsfsfd\",\"isActive\":\"true\"",
      "@version" : "1",
      "@timestamp" : "2020-01-26T19:17:49.977Z"
    }
  }

But, I wanted to get the message in json format as output.
{
"_index" : "test-product-logstash-22",
"_type" : "_doc",
"_id" : "10066",
"_score" : 1.0,
"_source" : {
"id" : "10066",
"prdctId" : "10066",
"name" : "ComPass",
"status" : "In Development",
"family" : "Strategic Growth",
"line" : "Strategic Growth",
"shortDesc" : "providers",
"longDesc" : "population",
"owner" : {
"id" : "E040547",
"name" : "Shashi "
},
"program" : {
"id" : "422",
"name" : "Regional Solutions and Bundles"
},
"alias" : [
{
"id" : "995c5dbd-97ee-46bb-b0d8-3e83a097762b",
"name" : "Community Pass"
}
],
"regAvail" : [
"ap",
"lac",
"mea"
],
"cntryAvail" : [
"n/a"
],
"salesCenterUrl" : "n/a",
"isActive" : "true"
}
}

Please suggest the filter needed to be applied to get the output in json format as shown above.

Use a json filter.

json { source => "message" }

Please have a look at the output received after applying json filter on message.
{
"message" => "{"prdctId":"10077","id":"10077","name":"ComPass","status":"In Development","family":"Strategic Growth","line":"Strategic Growth","shortDesc":"Enables interoperable digital IDs","longDesc":"A platform that enables interoperable digital ID's and other capabilities ","owner":{"id":"E040547","name":"Shashi Raghunandan"},"program":{"id":"422","name":"Regional Solutions and Bundles"},"alias":[{"id":"a1Id","name":"a1Name"},{"id":"a2Id","name":"a2Name"}],"regAvail":["ap","lac","mea"],"cntryAvail":["Global"],"salesCenterUrl":"salesCenterUrlsfsfd","isActive":"true"",
"tags" => [
[0] "_jsonparsefailure"
]
}

Please suggest.

Tried the below filter but then missing } at the end which hence not making message structure as json.
mutate { gsub => [ "message", "^[^{]+", "", "message", "}[^}]+$", "" ] }

Try

mutate { gsub => [ "message", "^[^{]+", "", "message", "[^}]+$", "" ] }

Thanks a lot. It's working fine now. After putting json filter, getting the proper structured message in output.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.