Dear Elastic folks,
We have a complex JSON data with multiple nested JSON arrays need to send to elastic search, Basically, we want to split the JSON array.
In short, we want to split a single big event with multiple arrays/lists to the separate events equal to numbers of arrays /lists, and also need to then split those each separate event.
Could you please provide a Logastsh pipeline to fulfill my request?
Raw JSON file
{
"Components": {
"msgprocessors": [
{
"processor": "eshop-retryOrder-fail-processor",
"active": true
},
{
"processor": "eshop-order-processor",
"active": true
},
{
"processor": "ch-retry-outbound-sampling",
"active": true
},
{
"processor": "pt-retry-outbound-sampling",
"active": true
},
{
"processor": "be-biometrics-inbox-processor",
"active": true
},
{
"processor": "pt-biometrics-inbox-processor",
"active": true
},
{
"processor": "ch-biometrics-inbox-processor",
"active": true
},
{
"processor": "uk-biometrics-inbox-processor",
"active": true
},
{
"processor": "eshop-retryAppointment-fail-processor",
"active": true
},
{
"processor": "be-retry-outbound-sampling",
"active": true
},
{
"processor": "uk-retry-outbound-sampling",
"active": true
},
{
"processor": "eshop-appointment-processor",
"active": true
},
{
"processor": "uk-priority-retry-outbound-sampling",
"active": true
}
],
"endpoints": [
{
"endpoint": "activeprocessor-admin-serviceEP",
"active": true
},
{
"endpoint": "api-admin-serviceEP",
"active": true
},
{
"endpoint": "be-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "ch-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "drools_antifraud",
"active": true
},
{
"endpoint": "drools_appointment_lateness",
"active": true
},
{
"endpoint": "drools_bioqueue",
"active": true
},
{
"endpoint": "drools_dynamic_endpoint",
"active": true
},
{
"endpoint": "endpoint-admin-serviceEP",
"active": true
},
{
"endpoint": "eshop-out-appointment-endpoint",
"active": true
},
{
"endpoint": "eshop-out-order-endpoint",
"active": true
},
{
"endpoint": "eshop_token_endpoint",
"active": true
},
{
"endpoint": "lb-be-encrypt-endpoint",
"active": true
},
{
"endpoint": "msgprocessor-admin-serviceEP",
"active": true
},
{
"endpoint": "pt-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "queue-admin-serviceEP",
"active": true
},
{
"endpoint": "servicegroup-admin-serviceEP",
"active": true
},
{
"endpoint": "tlsconnect_appointment_endpoint",
"active": true
},
{
"endpoint": "tlsconnect_order_endpoint",
"active": true
},
{
"endpoint": "tlsconnect_tma",
"active": true
},
{
"endpoint": "uk-filedecrypt-service-endpoint",
"active": true
},
{
"endpoint": "uk-hone1adapter-outbound-endpoint",
"active": true
}
],
"services": [
{
"service": "pt-filedecrypt-service",
"active": true
},
{
"service": "uk-file-processor-service",
"active": true
},
{
"service": "uk-filedecrypt-service",
"active": true
},
{
"service": "uk-hone1adapter-outbound-service",
"active": true
},
{
"service": "Version",
"active": true
}
],
"apis": [
{
"api": "appointmentLateness",
"active": true
},
{
"api": "bioqueue",
"active": true
},
{
"api": "pt-emetrics_TLS2PortugalAPI",
"active": true
},
{
"api": "order",
"active": true
},
{
"api": "be_token",
"active": true
},
{
"api": "GenericRulesEngineAPI",
"active": true
},
{
"api": "tma",
"active": true
},
{
"api": "sre",
"active": true
},
{
"api": "esbHealthCheckAPI",
"active": true
}
]
}
}
First, we want to split up four separate events for this single event above I posted.
we have 4 nested arrays/lists (maybe it's dynamically )for this case, they're :
[Components][msgprocessors]
[Components][services]
[Components][apis]
[Components][endpoints]
after that, we want to split each array event to have separate events.
The results we want to have:
Array: [Components][msgprocessors ]
{"Components": {"msgprocessors":{"processor":"eshop-retryOrder-fail-processor","active": true}}
{"Components": {"msgprocessors":{"processor":"eshop-order-processor","active": true}}
...................
Array: [Components][services]
{"Components": {"services":{"service":"pt-filedecrypt-service","active": true}}
{"Components": {"services":{"service":"uk-file-processor-service","active": true}}
...................
Array:[Components][apis]
{"Components": {"apis":{"api":"appointmentLateness","active": true}}
{"Components": {"apis":{"api":"bioqueue","active": true}}
................
Array:[Components][endpoints]
{"Components": {"endpoints":{"endpoint":"tlsconnect_appointment_endpoint","active": true}}
{"Components": {"endpoints":{"endpoint":"tlsconnect_order_endpoint","active": true}}
................