Split HTTP Poller Responses to Multiple Documents using Split Filter

[sajid (Sajid Ansari)]

We have multiple healthcheck endpoint of different applications which give response in json about the health status of that application and its dependencies.
Below are the response of two of the hc URLs:

URL 1 Response:

{
	"status": "Healthy",
	"totalDuration": "00:00:00.0600465",
	"entries": {
		"SabreService": {
			"data": {},
			"duration": "00:00:00.0599756",
			"status": "Healthy"
		},
		"FpLegacyService": {
			"data": {},
			"duration": "00:00:00.0104401",
			"status": "Healthy"
		},
		"TIDInfoAPI": {
			"data": {},
			"duration": "00:00:00.0062772",
			"status": "Healthy"
		},
		"MiniFareruleAPI": {
			"data": {},
			"duration": "00:00:00.0077703",
			"status": "Healthy"
		},
		"PaymentAPI": {
			"data": {},
			"duration": "00:00:00.0192433",
			"status": "Healthy"
		},
		"CredentialAPI": {
			"data": {},
			"duration": "00:00:00.0030248",
			"status": "Healthy"
		}
	}
}

URL 2 Response:

{
	"status": "Healthy",
	"totalDuration": "00:00:00.0055164",
	"entries": {
		"redisHealthCheck-Redis": {
			"data": {},
			"duration": "00:00:00.0008787",
			"status": "Healthy"
		},
		"apiHealthCheck-DBProxy Health Check": {
			"data": {},
			"duration": "00:00:00.0014546",
			"status": "Healthy"
		},
		"apiHealthCheck-ContentRouter Health Check": {
			"data": {},
			"duration": "00:00:00.0031306",
			"status": "Healthy"
		}
	}
}

All the URLs give similar response and they will have different dependencies in "entries" section.
I want the responses to be split into multiple documents i.e one document for one dependency. For ex, if pick first URL response, it should be splitted into 6+1 document (6 dependencies and 1 with status and duration of the app itself) and for second URL it should be splitted into (3+1) docs.
Below is my logstash script. It is inserting the responses of all URLs but not splitting. I need help regarding this how can I achieve my requirements.

    input {
        http_poller {
            urls => {
                something3 => {
                    method => get
                    url => "http://something3.xyz/healthcheck"
                }
                something2 => {
                    method => get
                    url => "http://something2.xyz/healthcheck"
                }
                something1 => {
                    method => get
                    url => "http://something1.xyz/healthcheck"
                }
                something0 => {
                    method => get
                    url => "http://something0.xyz/healthcheck"
                }
            }
            schedule=> {"every" => "30s"}
            request_timeout => 100
            codec => "json"
            metadata_target => "metadata"
        }
    }

    filter {
        split { field => "entries" }
    }

    output {
        elasticsearch {
        hosts => "http://elasticsearch-master.logging:9200"
        index => "hc-events-%{+YYYY.MM.dd}"
        ilm_policy => "7-days-default"
        user => "********"
        password => "********"
      }
    }

[sajid (Sajid Ansari)]

I was able to split after converting the hash to array.

filter {
    ruby {
        code => '
            dependency = []
            event.get("entries").each { |k, v|
                v["dep_name"] = k
                dependency << v
            }
            event.set("dependency", dependency)
        '
    }
    split { field => "dependency" }

It fulfilled my one of the requirements to split the response into different docs but I still have to achieve one more thing i.e. to create one more doc/event which will have all the data without splitting or the data which is out of "entries" block.
Requesting a help on this.

[Badger]

If you want another event then the easiest way is to have the split filter create it for you. You could try something like

    code => '
        dependency = []
        event.get("entries").each { |k, v|
            v["dep_name"] = k
            dependency << v
        }

        e = event.to_hash
        e.remove("entries")
        e["isSpecial"] = true
        dependency << e

        event.set("dependency", dependency)
    '

then after the split

if [dependency][isSpecial] {
    ruby {
        code => '
            event.remove("dependency").each { |k, v|
                event.set(k,v)
            }
        '
    }
}

[sajid (Sajid Ansari)]

Thank you @Badger for your help. It works.
However e.remove("entries") was giving exception "undefined method `remove' for #Hash:0x564e9f36" so I have removed it and added "remove_field => ["entries"]" in mutate filter at the last of filter section. Below is the final filter section of the config:

filter {
    ruby {
		code => '
			dependency = []
			event.get("entries").each { |k, v|
				v["dep_name"] = k
				dependency << v
			}

			e = event.to_hash
			e["isSpecial"] = true
			dependency << e

			event.set("dependency", dependency)
		'
    }
    split { field => "dependency" }
	if [dependency][isSpecial] {
		ruby {
			code => '
				event.remove("dependency").each { |k, v|
					event.set(k,v)
				}
			'
			}
	}
	mutate {
        remove_field => ["entries"]
      }
}