How to parse message with random data using logstash filter?

Elastic Stack Logstash
1

I have received the below message from the producer at Axon Topic with some random ASCII data.
"message" => "\u0000\u0000\u0000\u0002�\u0004C1�\u0001\u0002�\u0003\u0000\u0000\u0000\u0000A�\u0003{"prdctId":"10077","id":"10077","name":"ComPass","owner":{"id":"E040547","name":"Shashi Raghunandan"},"alias":[{"id":"a1Id","name":"a1Name"},{"id":"a2Id","name":"a2Name"}],"cntryAvail":["n/a"]}�����[\u0002\u0000\u0002\u0000"

I have tried split filter on open curly braces '{' to take out the outer json data from the message.
mutate {
split => ["message", "{"]
add_field => { "splitmsg" => "%{[message][0]}" }
add_field => { "msg" => "%{[message][1]}" }
}

but then we get the splitted messages from the message field, which seems not a correct way to do it as after that we got the message in below format.
{"prdctId":"10077","id":"10077","name":"ComPass","owner":{"id":"E040547","name":"Shashi Raghunandan"},"alias":[{"id":"a1Id","name":"a1Name"},{"id":"a2Id","name":"a2Name"}],"cntryAvail":["n/a"]}

Now, I'm not able to remove backslashes from the message. Please suggest the filter to be applied in such a way to have the output as json from above message into inline format.
{"prdctId":"10077","id":"10077","name":"ComPass","owner":{"id":"E040547","name":"Shashi Raghunandan"},"alias":[{"id":"a1Id","name":"a1Name"},{"id":"a2Id","name":"a2Name"}],"cntryAvail":["n/a"]}

2

You could try something like

mutate { gsub => [ "message", "^[^{]+", "", "message", "}[^}]+$", "" ] }

which would remove everything before the first { and everything after the last }.

3

After trying above filter, got message as given below.

{
    "_index" : "test-product-logstash-22",
    "_type" : "_doc",
    "_id" : "%{id}",
    "_score" : 1.0,
    "_source" : {
      "message" : "{\"prdctId\":\"10077\",\"id\":\"10077\",\"name\":\"ComPass\",\"status\":\"In Development\",\"family\":\"Strategic Growth\",\"line\":\"Strategic Growth\",\"shortDesc\":\"Enables interoperable digital IDs\",\"longDesc\":\"A platform that enables interoperable digital ID's and other capabilities \",\"owner\":{\"id\":\"E040547\",\"name\":\"Shashi Raghunandan\"},\"program\":{\"id\":\"422\",\"name\":\"Regional Solutions and Bundles\"},\"alias\":[{\"id\":\"a1Id\",\"name\":\"a1Name\"},{\"id\":\"a2Id\",\"name\":\"a2Name\"}],\"regAvail\":[\"ap\",\"lac\",\"mea\"],\"cntryAvail\":[\"Global\"],\"salesCenterUrl\":\"salesCenterUrlsfsfd\",\"isActive\":\"true\"",
      "@version" : "1",
      "@timestamp" : "2020-01-26T19:17:49.977Z"
    }
  }

But, I wanted to get the message in json format as output.
{
"_index" : "test-product-logstash-22",
"_type" : "_doc",
"_id" : "10066",
"_score" : 1.0,
"_source" : {
"id" : "10066",
"prdctId" : "10066",
"name" : "ComPass",
"status" : "In Development",
"family" : "Strategic Growth",
"line" : "Strategic Growth",
"shortDesc" : "providers",
"longDesc" : "population",
"owner" : {
"id" : "E040547",
"name" : "Shashi "
},
"program" : {
"id" : "422",
"name" : "Regional Solutions and Bundles"
},
"alias" : [
{
"id" : "995c5dbd-97ee-46bb-b0d8-3e83a097762b",
"name" : "Community Pass"
}
],
"regAvail" : [
"ap",
"lac",
"mea"
],
"cntryAvail" : [
"n/a"
],
"salesCenterUrl" : "n/a",
"isActive" : "true"
}
}

Please suggest the filter needed to be applied to get the output in json format as shown above.

4

Use a json filter.

json { source => "message" }
5

Please have a look at the output received after applying json filter on message.
{
"message" => "{"prdctId":"10077","id":"10077","name":"ComPass","status":"In Development","family":"Strategic Growth","line":"Strategic Growth","shortDesc":"Enables interoperable digital IDs","longDesc":"A platform that enables interoperable digital ID's and other capabilities ","owner":{"id":"E040547","name":"Shashi Raghunandan"},"program":{"id":"422","name":"Regional Solutions and Bundles"},"alias":[{"id":"a1Id","name":"a1Name"},{"id":"a2Id","name":"a2Name"}],"regAvail":["ap","lac","mea"],"cntryAvail":["Global"],"salesCenterUrl":"salesCenterUrlsfsfd","isActive":"true"",
"tags" => [
[0] "_jsonparsefailure"
]
}

Please suggest.

6

Tried the below filter but then missing } at the end which hence not making message structure as json.
mutate { gsub => [ "message", "^[^{]+", "", "message", "}[^}]+$", "" ] }

7

Try

mutate { gsub => [ "message", "^[^{]+", "", "message", "[^}]+$", "" ] }

  1. ^{ ↩︎

8

Thanks a lot. It's working fine now. After putting json filter, getting the proper structured message in output.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.