Hello everyone,
I am trying to parse a json document using logstash version 8.3.3. The json document has multiple nested arrays, to flatten the document split is being used inside the filter. The issue is that the splits are taking too long to complete almost 15-20 mins for 10 documents and most of the time this also hangs logstash. I have also tried using multiple workers without any luck. The system running logstash has 24 GB dedicated to logstash JVM. Does anyone know of a better method to parse the document properly, maybe without using split? (Attaching sample document and filter being used).
Sample Data:
{ "AppManager-response": { "result": { "response": { "Monitorinfo": { "Attribute": [ "", "", "", "", "", "", "", "", "", "", "", "", "", "" ], "CHILDMONITORS": [ { "CHILDMONITORINFO": [ { "CHILDATTRIBUTES": [ "", "", "", "" ] }, { "CHILDATTRIBUTES": [ "", "", "", "" ] } ] }, { "CHILDMONITORINFO": [ { "CHILDATTRIBUTES": [ "", "", "", "" ] }, { "CHILDATTRIBUTES": [ "", "", "", "" ] } ] } ] } } } } }
Filter Used:
filter{
split{field => "[result][response][0][Attribute]"}
split{field => "[result][response][0][CHILDMONITORS]", target => "splitjson"}
split{field => "[splitjson][CHILDMONITORS]", target => "nestedsplitjson"}
split{field => "[nestedsplitjson][CHILDATTRIBUTES]", target => "finalsplitjson"}
}
Kindly assist.