# Split HTTP Poller Responses to Multiple Documents using Split Filter

**URL:** <https://discuss.elastic.co/t/split-http-poller-responses-to-multiple-documents-using-split-filter/318418>\
**Category:** Logstash\
**Created:** [November 8, 2022, 10:40am UTC](https://discuss.elastic.co/t/split-http-poller-responses-to-multiple-documents-using-split-filter/318418 "2022-11-08T10:40:59Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sajid](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sajid](https://discuss.elastic.co/u/sajid)\
**Post date:** [November 8, 2022, 10:40am UTC](https://discuss.elastic.co/t/split-http-poller-responses-to-multiple-documents-using-split-filter/318418/1 "2022-11-08T10:40:59Z")

</div>

We have multiple healthcheck endpoint of different applications which give response in json about the health status of that application and its dependencies.  
Below are the response of two of the hc URLs:

**URL 1 Response:**

```auto
{
	"status": "Healthy",
	"totalDuration": "00:00:00.0600465",
	"entries": {
		"SabreService": {
			"data": {},
			"duration": "00:00:00.0599756",
			"status": "Healthy"
		},
		"FpLegacyService": {
			"data": {},
			"duration": "00:00:00.0104401",
			"status": "Healthy"
		},
		"TIDInfoAPI": {
			"data": {},
			"duration": "00:00:00.0062772",
			"status": "Healthy"
		},
		"MiniFareruleAPI": {
			"data": {},
			"duration": "00:00:00.0077703",
			"status": "Healthy"
		},
		"PaymentAPI": {
			"data": {},
			"duration": "00:00:00.0192433",
			"status": "Healthy"
		},
		"CredentialAPI": {
			"data": {},
			"duration": "00:00:00.0030248",
			"status": "Healthy"
		}
	}
}

```

**URL 2 Response:**

```auto
{
	"status": "Healthy",
	"totalDuration": "00:00:00.0055164",
	"entries": {
		"redisHealthCheck-Redis": {
			"data": {},
			"duration": "00:00:00.0008787",
			"status": "Healthy"
		},
		"apiHealthCheck-DBProxy Health Check": {
			"data": {},
			"duration": "00:00:00.0014546",
			"status": "Healthy"
		},
		"apiHealthCheck-ContentRouter Health Check": {
			"data": {},
			"duration": "00:00:00.0031306",
			"status": "Healthy"
		}
	}
}

```

All the URLs give similar response and they will have different dependencies in "entries" section.  
I want the responses to be split into multiple documents i.e one document for one dependency. For ex, if pick first URL response, it should be splitted into 6+1 document (6 dependencies and 1 with status and duration of the app itself) and for second URL it should be splitted into (3+1) docs.  
Below is my logstash script. It is inserting the responses of all URLs but not splitting. I need help regarding this how can I achieve my requirements.

```auto
    input {
        http_poller {
            urls => {
                something3 => {
                    method => get
                    url => "http://something3.xyz/healthcheck"
                }
                something2 => {
                    method => get
                    url => "http://something2.xyz/healthcheck"
                }
                something1 => {
                    method => get
                    url => "http://something1.xyz/healthcheck"
                }
                something0 => {
                    method => get
                    url => "http://something0.xyz/healthcheck"
                }
            }
            schedule=> {"every" => "30s"}
            request_timeout => 100
            codec => "json"
            metadata_target => "metadata"
        }
    }

    filter {
        split { field => "entries" }
    }

    output {
        elasticsearch {
        hosts => "http://elasticsearch-master.logging:9200"
        index => "hc-events-%{+YYYY.MM.dd}"
        ilm_policy => "7-days-default"
        user => " ********"
        password => " ********"
      }
    }

```

---

<div class="post-metadata">

**Author:** ![sajid](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sajid](https://discuss.elastic.co/u/sajid)\
**Post date:** [November 8, 2022, 4:01pm UTC](https://discuss.elastic.co/t/split-http-poller-responses-to-multiple-documents-using-split-filter/318418/2 "2022-11-08T16:01:23Z")

</div>

I was able to split after converting the hash to array.

```auto
filter {
    ruby {
        code => '
            dependency = []
            event.get("entries").each { |k, v|
                v["dep_name"] = k
                dependency << v
            }
            event.set("dependency", dependency)
        '
    }
    split { field => "dependency" }

```

It fulfilled my one of the requirements to split the response into different docs but I still have to achieve one more thing i.e. to create one more doc/event which will have all the data without splitting or the data which is out of "entries" block.  
Requesting a help on this.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 8, 2022, 7:26pm UTC](https://discuss.elastic.co/t/split-http-poller-responses-to-multiple-documents-using-split-filter/318418/3 "2022-11-08T19:26:58Z")

</div>

If you want another event then the easiest way is to have the split filter create it for you. You could try something like

```
    code => '
        dependency = []
        event.get("entries").each { |k, v|
            v["dep_name"] = k
            dependency << v
        }

        e = event.to_hash
        e.remove("entries")
        e["isSpecial"] = true
        dependency << e

        event.set("dependency", dependency)
    '

```

then after the split

```
if [dependency][isSpecial] {
    ruby {
        code => '
            event.remove("dependency").each { |k, v|
                event.set(k,v)
            }
        '
    }
}

```

---

<div class="post-metadata">

**Author:** ![sajid](https://avatars.discourse-cdn.com/v4/letter/s/779978/32.png) [@sajid](https://discuss.elastic.co/u/sajid)\
**Post date:** [November 8, 2022, 8:33pm UTC](https://discuss.elastic.co/t/split-http-poller-responses-to-multiple-documents-using-split-filter/318418/4 "2022-11-08T20:33:14Z")

</div>

Thank you @Badger for your help. It works.  
However e.remove("entries") was giving exception "undefined method `remove' for #Hash:0x564e9f36" so I have removed it and added "remove\_field =\> ["entries"]" in mutate filter at the last of filter section. Below is the final filter section of the config:

```auto
filter {
    ruby {
		code => '
			dependency = []
			event.get("entries").each { |k, v|
				v["dep_name"] = k
				dependency << v
			}

			e = event.to_hash
			e["isSpecial"] = true
			dependency << e

			event.set("dependency", dependency)
		'
    }
    split { field => "dependency" }
	if [dependency][isSpecial] {
		ruby {
			code => '
				event.remove("dependency").each { |k, v|
					event.set(k,v)
				}
			'
			}
	}
	mutate {
        remove_field => ["entries"]
      }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 6, 2022, 8:33pm UTC](https://discuss.elastic.co/t/split-http-poller-responses-to-multiple-documents-using-split-filter/318418/5 "2022-12-06T20:33:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.