Logstash- http_poller: Filter need to execute after getting first and second URL request’s response

I have two URLs and i need execute filters functionality after getting the responses of each . Because i need merge the response of these url response and parse responses and put it into the elastic search.But when i tried to execute my conf file first http response is putting into elastic and then second response data .
Can you please help me to merge (aggregate) the result into elastic search.

Due to security concern i will share dummy conf file.

My input like this:

input {
http_poller {
schedule => { cron => "* /1 * * * UTC"}
# List of urls to hit
# URLs can either have a simple format for a get request
# Or use more complex HTTP features
urls => {
requestServiceA => {
codec => "plain"
method => "post"
url => "http://jsbadab"
headers => {
Accept => "application/xml"
}

  }

requestServiceB => {
codec => "plain"
method => "post"
url => "http://sdksndfik"
headers => {
Accept => "application/xml"
}

}
# Maximum amount of time to wait for a request to complete
request_timeout => 45
# FacilityLisrequestPortServicetHow far apart requests should be
#interval => 60
# Decode the results as xml
codec => "plain"
# Store metadata about the request in this key
metadata_target => "http_poller_metadata"

}

}

My filter like this:

filter {

# if [http_poller_metadata]{
	mutate {
		add_field => { "requestName" => "%{http_poller_metadata[name]}" }
		#add_field => {"requestName" =>  "%{http_poller_metadata_port [name]}" } 
	 }
# }

 
 if [requestName]=="requestServiceA" {
  xml {
	#add_field => {"abcd" => "abcdef"}
	source => "message"
	target => "aResponse"
	remove_field => ["http_poller_metadata","@version","@timestamp","message"]    
  }
  
  
 
  
}

if [requestName]=="requestServiceB"{
xml {
# add_field => {"abc" => "abcdef"}
source => "message"
target => "bResponse"
remove_field => ["http_poller_metadata","@version","@timestamp"]
}

}

My output like this:

{
"took": 0,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "test30",
"_type": "test",
"_id": "AWVcFK94TgnFfJZNSEmw",
"_score": 1,
"_source": {
"aResponse":[]
}
},
{
"_index": "test30",
"_type": "test",
"_id": "AWVcFLAWTgnFfJZNSEmx",
"_score": 1,
"_source": {
"bResponse":[]
}
}
]
}
}

Expecting out put like this:

{
"took": 0,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "test30",
"_type": "test",
"_id": "AWVcFK94TgnFfJZNSEmw",
"_score": 1,
"_source": {
"aResponse":[],
"bResponse":[]
}
}

    ]
}

}

It is very urgent.Can you please response ASAP.Thanks in advance.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.