Help to handle the nested JSON array

Elastic Stack Logstash
1

Dear Elastic folks,

We have a complex JSON data with multiple nested JSON arrays need to send to elastic search, Basically, we want to split the JSON array.
In short, we want to split a single big event with multiple arrays/lists to the separate events equal to numbers of arrays /lists, and also need to then split those each separate event.

Could you please provide a Logastsh pipeline to fulfill my request?

Raw JSON file

{
  "Components": {
    "msgprocessors": [
      {
        "processor": "eshop-retryOrder-fail-processor",
        "active": true
      },
      {
        "processor": "eshop-order-processor",
        "active": true
      },
      {
        "processor": "ch-retry-outbound-sampling",
        "active": true
      },
      {
        "processor": "pt-retry-outbound-sampling",
        "active": true
      },
      {
        "processor": "be-biometrics-inbox-processor",
        "active": true
      },
      {
        "processor": "pt-biometrics-inbox-processor",
        "active": true
      },
      {
        "processor": "ch-biometrics-inbox-processor",
        "active": true
      },
      {
        "processor": "uk-biometrics-inbox-processor",
        "active": true
      },
      {
        "processor": "eshop-retryAppointment-fail-processor",
        "active": true
      },
      {
        "processor": "be-retry-outbound-sampling",
        "active": true
      },
      {
        "processor": "uk-retry-outbound-sampling",
        "active": true
      },
      {
        "processor": "eshop-appointment-processor",
        "active": true
      },
      {
        "processor": "uk-priority-retry-outbound-sampling",
        "active": true
      }
    ],
    "endpoints": [
      {
        "endpoint": "activeprocessor-admin-serviceEP",
        "active": true
      },
      {
        "endpoint": "api-admin-serviceEP",
        "active": true
      },
      {
        "endpoint": "be-filedecrypt-service-endpoint",
        "active": true
      },
      {
        "endpoint": "ch-filedecrypt-service-endpoint",
        "active": true
      },
      {
        "endpoint": "drools_antifraud",
        "active": true
      },
      {
        "endpoint": "drools_appointment_lateness",
        "active": true
      },
      {
        "endpoint": "drools_bioqueue",
        "active": true
      },
      {
        "endpoint": "drools_dynamic_endpoint",
        "active": true
      },
      {
        "endpoint": "endpoint-admin-serviceEP",
        "active": true
      },
      {
        "endpoint": "eshop-out-appointment-endpoint",
        "active": true
      },
      {
        "endpoint": "eshop-out-order-endpoint",
        "active": true
      },
      {
        "endpoint": "eshop_token_endpoint",
        "active": true
      },
      {
        "endpoint": "lb-be-encrypt-endpoint",
        "active": true
      },
      {
        "endpoint": "msgprocessor-admin-serviceEP",
        "active": true
      },
      {
        "endpoint": "pt-filedecrypt-service-endpoint",
        "active": true
      },
      {
        "endpoint": "queue-admin-serviceEP",
        "active": true
      },
      {
        "endpoint": "servicegroup-admin-serviceEP",
        "active": true
      },
      {
        "endpoint": "tlsconnect_appointment_endpoint",
        "active": true
      },
      {
        "endpoint": "tlsconnect_order_endpoint",
        "active": true
      },
      {
        "endpoint": "tlsconnect_tma",
        "active": true
      },
      {
        "endpoint": "uk-filedecrypt-service-endpoint",
        "active": true
      },
      {
        "endpoint": "uk-hone1adapter-outbound-endpoint",
        "active": true
      }
    ],
    "services": [
      {
        "service": "pt-filedecrypt-service",
        "active": true
      },
      {
        "service": "uk-file-processor-service",
        "active": true
      },
      {
        "service": "uk-filedecrypt-service",
        "active": true
      },
      {
        "service": "uk-hone1adapter-outbound-service",
        "active": true
      },
      {
        "service": "Version",
        "active": true
      }
    ],
    "apis": [
      {
        "api": "appointmentLateness",
        "active": true
      },
      {
        "api": "bioqueue",
        "active": true
      },
      {
        "api": "pt-emetrics_TLS2PortugalAPI",
        "active": true
      },
      {
        "api": "order",
        "active": true
      },
      {
        "api": "be_token",
        "active": true
      },
      {
        "api": "GenericRulesEngineAPI",
        "active": true
      },
      {
        "api": "tma",
        "active": true
      },
      {
        "api": "sre",
        "active": true
      },
      {
        "api": "esbHealthCheckAPI",
        "active": true
      }
    ]
  }
}

First, we want to split up four separate events for this single event above I posted.

we have 4 nested arrays/lists (maybe it's dynamically )for this case, they're :

[Components][msgprocessors]
[Components][services]
[Components][apis]
[Components][endpoints]

​after that, we want to split each array event to have separate events.

The results we want to have:

Array: [Components][msgprocessors ]

{"Components": {"msgprocessors":{"processor":"eshop-retryOrder-fail-processor","active": true}}
{"Components": {"msgprocessors":{"processor":"eshop-order-processor","active": true}}
...................

Array: [Components][services]

{"Components": {"services":{"service":"pt-filedecrypt-service","active": true}}
{"Components": {"services":{"service":"uk-file-processor-service","active": true}}
...................

Array:[Components][apis]

{"Components": {"apis":{"api":"appointmentLateness","active": true}}
{"Components": {"apis":{"api":"bioqueue","active": true}}
................

Array:[Components][endpoints]

{"Components": {"endpoints":{"endpoint":"tlsconnect_appointment_endpoint","active": true}}
{"Components": {"endpoints":{"endpoint":"tlsconnect_order_endpoint","active": true}}
................
2

You may need to combine split and clone filter

3

I do my best, i couldn't implement it.
Would you please help me out?

Thanks in advance

4

Any updates?
by the way we're using logstash 6.4.
Thanks

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.