How to parse multiple nested arrays

Elastic Stack Logstash
1

Hello everyone,

I am trying to parse a json document using logstash version 8.3.3. The json document has multiple nested arrays, to flatten the document split is being used inside the filter. The issue is that the splits are taking too long to complete almost 15-20 mins for 10 documents and most of the time this also hangs logstash. I have also tried using multiple workers without any luck. The system running logstash has 24 GB dedicated to logstash JVM. Does anyone know of a better method to parse the document properly, maybe without using split? (Attaching sample document and filter being used).

Sample Data:

{
  "AppManager-response": {
    "result": {
      "response": {
        "Monitorinfo": {
          "Attribute": [
            "",
            "",
            "",
            "",
            "",
            "",
            "",
            "",
            "",
            "",
            "",
            "",
            "",
            ""
          ],
          "CHILDMONITORS": [
            {
              "CHILDMONITORINFO": [
                {
                  "CHILDATTRIBUTES": [
                    "",
                    "",
                    "",
                    ""
                  ]
                },
                {
                  "CHILDATTRIBUTES": [
                    "",
                    "",
                    "",
                    ""
                  ]
                }
              ]
            },
            {
              "CHILDMONITORINFO": [
                {
                  "CHILDATTRIBUTES": [
                    "",
                    "",
                    "",
                    ""
                  ]
                },
                {
                  "CHILDATTRIBUTES": [
                    "",
                    "",
                    "",
                    ""
                  ]
                }
              ]
            }
          ]
        }
      }
    }
  }
}

Filter Used:

filter{

split{field => "[result][response][0][Attribute]"}
split{field => "[result][response][0][CHILDMONITORS]", target => "splitjson"}
split{field => "[splitjson][CHILDMONITORS]", target => "nestedsplitjson"}
split{field => "[nestedsplitjson][CHILDATTRIBUTES]", target => "finalsplitjson"}

}

Kindly assist.

2

If you look at the code you will see that if the split value is empty then it is discarded. So when you do the split on

    "Monitorinfo": {
      "Attribute": [
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        ""
      ],

the split filter breaks it up into 14 parts, discards them all, and then cancels the event it split.

3

Thanks for your response, and sorry for the confusion. Unfortunately I will not be able to share the complete data set as it contains sensitive information hence I have only shared the blank template. I am getting data inside these arrays. Do you know of any other way of splitting these nested arrays?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.