How to get the nested array and nested json from input message in logstash

Elastic Stack Logstash
1

message : {"prdctId":"10066","id":"10066","owner":{"id":"E040547","name":"Shashi Raghunandan"},"regAvail":["ap","lac","mea"]
,"isActive":true}

Applied Filter:
kv {
source => "message"
trim_key => """
trim_value => """
field_split => ","
value_split => ":"
include_keys => [ "prdctId","id", "owner", "name", "isActive", "regAvail" ]
}
mutate {
remove_field => ["splitmsg", "@version", "@timestamp", "message"]
}

Output:
The response coming with field names - prdctId, id, name, isActive but not showing for owner since it is a nested json, also for regAvail which is itself a nested array.
{
"_id" : "10066",
"_score" : 1.0,
"_source" : {
"name" : "ComPass",
"prdctId" : "10066",
"isAvtive" : true,
"id" : "10066"
}
}
Please let me know what filter should be applied to get owner nested json and regAvail nested array in output.

2

Hi there,

it is as simple as

filter {
  json {
    source => "message"
  }
}
1 Like
3

this filter didn't worked out for us. I am getting the output with backslash as well.
indent preformatted text by 4 spaces
what is the way to remove it ... tried lots of stuff but none of them is worked out for me."msg" : "\"prdctId\":\"10077\",\"id\":\"10077\",\"name\":\"ComPass\",\"owner\":\"id\":\"E040547\",\"name\":\"Shashi Raghunandan\",\"alias\":[{\"id\":\"a1\",\"name\":\"a1\",{\"id\":\"a2\",\"name\":\"a2\"],\"cntryAvail\":[\"n/a\"]"

4

Hi

If the fields in your message

are known and fixed, you could access them "manually", like so:

mutate {
        add_field => {
          "PRDCTLD" => "%{[message][prdctld]}"
          "ID" => "%{[message][id]}"
          "OWNER_ID" => "%{[message][owner][id]}"
          "OWNER_NAME" => "%{[message][owner][name]}"
          "REGAVAIL_0" =>  "%{[message][regAvail][0]}"
          "REGAVAIL_1" =>  "%{[message][regAvail][1]}"
          "REGAVAIL_2" =>  "%{[message][regAvail][2]}"
          "ISACTIVE" => "%{[message][isActive]}"
        }
}

Check the syntax, just in case, but it should work with minor tweaks.

Hope this helps.

5

You can use either the JSON filter plugin or the JSON input codec to transform the input text into a JSON object.

6

NO, we are not able to remove backslash from the string .
Tried
mutate { gsub => [ "message", "[\]", "" ] }
but didn't found it working.

Actually, we are getting input as given below from which we are trying to fetch the json string starting from { to }.
"message" => "\u0000\u0000\u0000\u0002�\u0004C1�\u0001CorporateBusinessTechnology.InternalServices-B2EServices.TestProductStr\u0002�\u0003\u0000\u0000\u0000\u0000A�\u0003{"prdctId":"10077","id":"10077","name":"ComPass","owner":{"id":"E040547","name":"Shashi Raghunandan"},"alias":[{"id":"a1Id","name":"a1Name"},{"id":"a2Id","name":"a2Name"}],"cntryAvail":["n/a"]}�����[\u0002\u0000\u0002\u0000"

Please suggest filter to be used to get the json from message.

7

So, if you manage to first extract the json part you're interested in and put it in a field (e.g. json_field) and then do something like

mutate {
  gsub => [
    "json_field", "\n", "\\n",
    "json_field", "\t", "\\t",
    "json_field", "\r", "\\r"
  ]
}

json {
  source => "json_field"
}

will it work?

8

Anyway, can you please share the output of a simple pipeline

input {
  whatever input you're using
}

filter {}

output {
  stdout {}
}

I want to see how logstash receive and read the data.

9

Hi

Whould you care to post your entire code? It would be helpful if we could see your input{} and filter{} sections.

It would also be great if we could see

  • a sample of your intput data
  • the output (stdout{}) from your sample data
  • the output (stdout{}) from your sample data with no filter{} section (just comment out the whole filter{} thing and run logstash once).
  • any log entries that might be relevant

Please, post all of it in one post, organized so there's no confusion as to what goes with what. Try to be as clear, detailed and specific as you can.

Thank you.

1 Like
10

Yes but I am getting this as an output now.
Along with an error message - "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [owner] of type [text] in document with id '10077'. Preview of field's value: '{name=Shashi Raghunandan, id=E040547}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:89"}}}}}

{
     "shortDesc" => "Enables interoperable digital IDs",
         "alias" => [
    [0] {
        "name" => "a1Name",
          "id" => "a1Id"
    },
    [1] {
        "name" => "a2Name",
          "id" => "a2Id"
    }
],
      "regAvail" => [
    [0] "ap",
    [1] "lac",
    [2] "mea"
],
        "family" => "Strategic Growth",
"salesCenterUrl" => "n/a",
      "isActive" => true,
       "prdctId" => "10077",
          "line" => "Strategic Growth",
          "name" => "ComPass",
         "owner" => {
    "name" => "Shashi Raghunandan",
      "id" => "E040547"
},
      "longDesc" => "A platform that enables interoperable digital ID's and other capabilities ",
    "cntryAvail" => [
    [0] "n/a"
],
        "status" => "In Development",
            "id" => "10077"

}

but my requirement is to have inline output.
{
"shortDesc" => "Enables interoperable digital IDs",
"alias" => [
{
"name" => "a1Name",
"id" => "a1Id"
},
{
"name" => "a2Name",
"id" => "a2Id"
}
],
"regAvail" => [
"ap",
"lac",
"mea"
],
"family" => "Strategic Growth",
"salesCenterUrl" => "n/a",
"isActive" => true,
"prdctId" => "10077",
"line" => "Strategic Growth",
"name" => "ComPass",
"owner" => {
"name" => "Shashi Raghunandan",
"id" => "E040547"
},
"longDesc" => "A platform that enables interoperable digital ID's and other capabilities ",
"cntryAvail" => [
"n/a"
],
"status" => "In Development",
"id" => "10077"
}

Also, for your reference, please have a look at the filter placed on input message.
filter {
mutate {
split => ["message", "#"]
add_field => { "firstrandomdata" => "%{[message][0]}" }
add_field => { "msg" => "%{[message][1]}" }
add_field => { "secondrandomdata" => "%{[message][2]}" }
}
mutate {
rename => ["msg", "message" ]
}
mutate {
gsub => [
"message", "\n", "\n",
"message", "\t", "\t",
"message", "\r", "\r"
]
}
json {
source => "message"
}
mutate {
remove_field => ["@version", "@timestamp", "firstrandomdata", "secondrandomdata", "message"]
}
}

11

Appreciate if any suggestion posted for the query.

12

Listen, you're mixing everything generating a bit of confusion in us (at least in me).
You reported output messages with fields family and salesCenterUrl which are not present in any of your input messages.

You're talking about backslashes in the output without specifying where you're trying to store the output (Elasticsearch? A file? Whatever else?).

@ITIC kindly asked you to make a post describing EVERYTHING. Which means writing a post structured like:

------ SAMPLE OF INPUT DATA -----
....pasting your FORMATTED input here....

------ DATA PROCESSED BY A NO-FILTER LOGSTASH PIPELINE --------
....pasting your FORMATTED non filtered output here....

------ LOGSTASH PIPELINE WITH FILTER ------
....pasting your FORMATTED pipeline here....

------ DATA PROCESSED BY THE ABOVE LOGSTASH PIPELINE --------
....pasting your FORMATTED filtered output here....

------ DESIRED OUTPUT --------
....pasting your FORMATTED desired output here....

This way we can help you solving your problem. If you're the first one who first posts a message with an input and then posts a (filtered? unfiltered?) output with fields not corresponding to the ones in input, it'll be very difficult for us to help you.

Please, provide us with what I asked you in this post, so we can think of something.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.