I have two URLs and i need execute filters functionality after getting the responses of each . Because i need merge the response of these url response and parse responses and put it into the elastic search.But when i tried to execute my conf file first http response is putting into elastic and then second response data .
Can you please help me to merge (aggregate) the result into elastic search.
Due to security concern i will share dummy conf file.
My input like this:
input {
http_poller {
schedule => { cron => "* /1 * * * UTC"}
# List of urls to hit
# URLs can either have a simple format for a get request
# Or use more complex HTTP features
urls => {
requestServiceA => {
codec => "plain"
method => "post"
url => "http://jsbadab"
headers => {
Accept => "application/xml"
}
}
requestServiceB => {
codec => "plain"
method => "post"
url => "http://sdksndfik"
headers => {
Accept => "application/xml"
}
}
# Maximum amount of time to wait for a request to complete
request_timeout => 45
# FacilityLisrequestPortServicetHow far apart requests should be
#interval => 60
# Decode the results as xml
codec => "plain"
# Store metadata about the request in this key
metadata_target => "http_poller_metadata"
}
}
My filter like this:
filter {
# if [http_poller_metadata]{
mutate {
add_field => { "requestName" => "%{http_poller_metadata[name]}" }
#add_field => {"requestName" => "%{http_poller_metadata_port [name]}" }
}
# }
if [requestName]=="requestServiceA" {
xml {
#add_field => {"abcd" => "abcdef"}
source => "message"
target => "aResponse"
remove_field => ["http_poller_metadata","@version","@timestamp","message"]
}
}
if [requestName]=="requestServiceB"{
xml {
# add_field => {"abc" => "abcdef"}
source => "message"
target => "bResponse"
remove_field => ["http_poller_metadata","@version","@timestamp"]
}
}
My output like this:
{
"took": 0,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "test30",
"_type": "test",
"_id": "AWVcFK94TgnFfJZNSEmw",
"_score": 1,
"_source": {
"aResponse":[]
}
},
{
"_index": "test30",
"_type": "test",
"_id": "AWVcFLAWTgnFfJZNSEmx",
"_score": 1,
"_source": {
"bResponse":[]
}
}
]
}
}
Expecting out put like this:
{
"took": 0,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "test30",
"_type": "test",
"_id": "AWVcFK94TgnFfJZNSEmw",
"_score": 1,
"_source": {
"aResponse":[],
"bResponse":[]
}
}
]
}
}
It is very urgent.Can you please response ASAP.Thanks in advance.