# How to send http\_poller response data in elastic search

**URL:** <https://discuss.elastic.co/t/how-to-send-http-poller-response-data-in-elastic-search/235519>\
**Category:** Logstash\
**Created:** [June 3, 2020, 10:49am UTC](https://discuss.elastic.co/t/how-to-send-http-poller-response-data-in-elastic-search/235519 "2020-06-03T10:49:50Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pankaj\_Jadhav](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pankaj_jadhav/32/65896_2.png) [@Pankaj\_Jadhav](https://discuss.elastic.co/u/Pankaj_Jadhav)\
**Post date:** [June 3, 2020, 10:49am UTC](https://discuss.elastic.co/t/how-to-send-http-poller-response-data-in-elastic-search/235519/1 "2020-06-03T10:49:50Z")

</div>

Hi Guys,

can someone will help me to add http\_poller response in elastic.

my config file  
input{  
kafka {  
bootstrap\_servers =\> "172.20.188.11:9092,172.20.188.31:9092,172.20.188.30:9092"  
topics =\> ["DS\_Test"]  
decorate\_events =\> true  
group\_id =\> "ds\_test"  
codec =\> json  
max\_poll\_records =\> "100"  
consumer\_threads =\> 2  
session\_timeout\_ms =\> "30000"  
auto\_offset\_reset =\> "earliest"  
client\_id =\> "logstash-ingestion"  
}  
}  
input {  
http\_poller {  
urls =\> {  
soap\_request =\> {  
method =\> post  
url =\> "[https://dm-dev-authenticationservice-serviceapps-use-rg.azurewebsites.net/api/v1/Token](https://dm-dev-authenticationservice-serviceapps-use-rg.azurewebsites.net/api/v1/Token)"  
headers =\> {  
"Content-Type" =\> "application/json"  
}  
body =\> '{"ClientId":"sdmtSG9BOap6aLn5oxoaCrfLHQCIUMLF","ClientSecret":"SqN4ywxCu6ylvn7ecoViEWB2NlUk9hup0eBn5VquT0XutSXCh6VKXC82dREeGIi2","Audience":"[https://InventoryService/](https://InventoryService/)"}'  
}  
}  
request\_timeout =\> 60  
# Supports "cron", "every", "at" and "in" schedules by rufus scheduler  
schedule =\> { cron =\> "\* \* \* \* \* UTC"}  
codec =\> "json"  
# A hash of request metadata info (timing, response headers, etc.) will be sent here  
metadata\_target =\> "http\_poller\_metadata"  
}  
}  
filter {  
mutate { remove\_field =\> "@Version" }  
mutate { remove\_field =\> "@timestamp" }  
mutate { add\_field =\> { "[@metadata][index]" =\> "%{[eventHeader][bpcName]}_%{[eventHeader][applicationId]}"} }  
mutate { add\_field =\> { "[@metadata][ObjectCode]" =\> "%{[eventHeader][applicationId]}_%{[eventHeader][objectCode]}"} }  
mutate { add\_field =\> { "[@metadata][action]" =\> "%{[event][action]}"} }  
mutate { add\_field =\> { "[@metadata][applicationId]" =\> "%{[eventHeader][applicationId]}"} }  
mutate { add\_field =\> { "[@metadata][itemId]" =\> "%{[eventData][itemId]}"} }  
mutate { add\_field =\> { "[@metadata][isDeleted]" =\> "%{[eventData][isDeleted]}"} }  
mutate { gsub =\> ["[@metadata][action]","INSERT","index"] }  
mutate { gsub =\> ["[@metadata][action]","Insert","index"] }  
mutate { gsub =\> ["[@metadata][action]","UPDATE","update"] }  
mutate { gsub =\> ["[@metadata][action]","Update","update"] }  
mutate { gsub =\> ["[@metadata][action]","DELETE","delete"] }  
mutate { gsub =\> ["[@metadata][action]","Delete","delete"] }  
mutate { remove\_field =\> "[logstash]" }  
mutate { remove\_field =\> "[metadata]" }  
mutate { remove\_field =\> "[eventHeader]" }  
mutate { remove\_field =\> "[event]" }  
json{  
source =\> "message"  
}  
mutate{  
add\_field =\> { "status" =\> "%{[http\_poller\_metadata][code]}" }  
}  
mutate {  
copy =\> {  
"status" =\> "statuscode"  
}  
}  
ruby {  
code =\> '  
details\_hash = event.get("eventData")  
return if details\_hash.nil?  
details\_hash.each do |k, v|  
event.set(k,v)  
end  
event.remove("eventData")  
hash = event.to\_hash  
hash.each {|k,v|  
data = event.get(k)  
return if data.nil?  
event.set(k,v)  
}'  
}  
json{ source=\>"message" }  
}  
output{  
if [@metadata][applicationId] == "62"  
{  
elasticsearch {  
hosts =\> "[http://172.20.188.15:9200](http://172.20.188.15:9200)"  
user =\> "elastic"  
password =\> "changeme"  
index =\> "%{[@metadata][index]}"  
document\_id =\> "%{[@metadata][ObjectCode]}"  
action =\> "%{[@metadata][action]}"  
}  
}  
stdout { codec =\> rubydebug { metadata =\> true} }  
}

My resposne:-

{  
"normalizedText" =\> "",  
"@metadata" =\> {  
"action" =\> "index",  
"index" =\> "flowersfoods\_62",  
"kafka" =\> {  
"key" =\> nil,  
"offset" =\> 226,  
"consumer\_group" =\> "ds\_test",  
"topic" =\> "DS\_Test",  
"partition" =\> 0,  
"timestamp" =\> -1  
},  
"ObjectCode" =\> "62\_125",  
"applicationId" =\> "62",  
"itemId" =\> "125",  
"isDeleted" =\> "%{[eventData][isDeleted]}"  
},  
"IndexedOn" =\> "2020-06-03T15:56:42.2120593+05:30",  
"type" =\> "AS",  
"nounModifier" =\> "",  
"statuscode" =\> "%{[http\_poller\_metadata][code]}",  
"itemId" =\> 125,  
"field" =\> "itemId",  
"softDelete" =\> "Y",  
"appId" =\> 36,  
"@version" =\> "1",  
"contactcode" =\> 0,  
"text" =\> 90,  
"status" =\> "%{[http\_poller\_metadata][code]}"  
}  
{  
"statuscode" =\> "200",  
"expiresIn" =\> 86400,  
"@metadata" =\> {  
"action" =\> "%{[event][action]}",  
"index" =\> "%{[eventHeader][bpcName]}_%{[eventHeader][applicationId]}",  
"ObjectCode" =\> "%{[eventHeader][applicationId]}_%{[eventHeader][objectCode]}",  
"applicationId" =\> "%{[eventHeader][applicationId]}",  
"itemId" =\> "%{[eventData][itemId]}",  
"isDeleted" =\> "%{[eventData][isDeleted]}"  
},  
"@version" =\> "1",  
"http\_poller\_metadata" =\> {  
"request" =\> {  
"headers" =\> {  
"Content-Type" =\> "application/json"  
},  
"method" =\> "post",  
"body" =\> "{"ClientId":"sdmtSG9BOap6aLn5oxoaCrfLHQCIUMLF","ClientSecret":"SqN4ywxCu6ylvn7ecoViEWB2NlUk9hup0eBn5VquT0XutSXCh6VKXC82dREeGIi2","Audience":"[https://InventoryService/"}"](https://InventoryService/%22%7D%22),  
"url" =\> "[https://dm-dev-authenticationservice-serviceapps-use-rg.azurewebsites.net/api/v1/Token](https://dm-dev-authenticationservice-serviceapps-use-rg.azurewebsites.net/api/v1/Token)"  
},  
"response\_headers" =\> {  
"date" =\> "Wed, 03 Jun 2020 10:27:00 GMT",  
"set-cookie" =\> [  
[0] "ARRAffinity=58e912a579bcbdcd4d39cdfe9f22b26a8257c0436fbb6567ba405f84999f917f;Path=/;HttpOnly;[Domain=dm-dev-authenticationservice-serviceapps-use-rg.azurewebsites.net](http://Domain=dm-dev-authenticationservice-serviceapps-use-rg.azurewebsites.net)",  
[1] "ARRAffinity=e3f151f06fbd0f78471b3feb8433ea3f4f2a489a9d92ad613a1ca01018b4cf04;Path=/;HttpOnly;[Domain=dm-dev-authenticationservice-serviceapps-use-rg.azurewebsites.net](http://Domain=dm-dev-authenticationservice-serviceapps-use-rg.azurewebsites.net)"  
],  
"request-context" =\> "appId=cid-v1:7d56ed69-504b-4a83-bc46-7bb3b515ed95",  
"vary" =\> "Accept-Encoding",  
"content-type" =\> "application/json; charset=utf-8"  
},  
"code" =\> 200,  
"response\_message" =\> "OK",  
"times\_retried" =\> 0,  
"runtime\_seconds" =\> 0.977467,  
"name" =\> "soap\_request",  
"host" =\> "SEADEVDMLOGSTASH7x2"  
},  
"accessToken" =\> "VdZG9Nd+CMJA6UVBAxPSE5eFdNt3lfJc6CkbmWs1m20NhpAxlaw41lBMHByhfwBlzuwLAdal9wRVZbaQm9+F+k2PW2A60bufj631K1OEcrndEtrfrIP0kdPKqTDzcby0eSW_Koy2_Oi6CburrtuTFsYyRCP71Go+Url9yQ1o812dXPzV2JfhwLeEcFGTSX7ktsRm1N9vfPiEeTHdM1Q04lMUvPu30dvI+AEyN9npdyauz5JQmTUWkwfLR5nNJiUwwdHGkdrMw7U+LTUwiNGFKx3MxYdbOxoD5ZxwRobF3_TcYqeQ35xT4kbeVHqjy2yKgGySh5sjiKdKfTisT4tQZ+X3rQA34HdorE4zK3ioQzsrlPSSm1KSmdomIOqpvg9UEnFoXmwlPsukjLX5mcx5BsHpDLUkhFuIwSvr98QhN9zHIPMWF2q26neECb4D6cKhzMpjF_KIlNu4aTORij5tL3gw7tz_WWKkoRjELTHWhrpa4SMizK8eaNVWnxgwbfplO9T510Esvz24JI8h1KAP387TTHlxL1IHAZK9MCgo2_QJCwSZRCPdv9H0Ucf8vNw_Adlkb+KuxNDZCjzTX31pi3gsku1FVevtsEBTPMYrpyksVvAKK2i07Vnc4LXpdoLSBY8mt94Pgyoosr2eYLf2z0HRh7CMrf+0elG50rnhp_ShWHAwvshuxDgepJrnY_kFOLKDMa5SxyABLnLeaCAhsZwrUOdCAUORel8qUda9xkFb3wTgkoR4eph5xKqAftzhgjGECxnt5XTF0wmMdZKlRk9M0y6gHYH1IjTdB8AIS6_sEBhPiONMFHQWaq6hoVO9y_Q6BBXkHwClLD2sPIhmq7mK5hKk1nrpXD95RuepeSm+Y8BaoDUxmlnC9gynyJ+jBtlLt8bfoKbPOd9s_PKe5LtxC_2oeabB465mlQJerWbekUwsbfjNs_sOyixpI4H5_JCij8a4kI5Xo_Tleq2nYdWsRnDTDvxn7xAW6cOPk_IVb8hNxbkotVM9IdkboF4Ac1_ce2XS1PeN03387UVFz_is+sOLprWyG7JmCVQjqaluYzzeWnxaskJn2NBduy3Wz1L0gINzEv+siajVABwBWlzo432GfCbSCU+T+3gd8H9MtFtHWOINqQlqbvALA3MGwfqOtZzUIPNMREx3zxCCdNpDGMdExxXN_vq5_UhrLMgLR6ZRQuiB7fQnrArLHifBDBnbBnoBGTqbFzS_2fCKdji0XxrKfxNKQ2X+X2H718JOciE4SeuEBPuYeO8rhxBDBZPvOZKy2zQ9KqzNRdH9nIZrCXkr1MFrrgi1FioEWfEhirOimAXeYUOlsYlCHm",  
"tokenType" =\> "Bearer",  
"status" =\> "200"  
}

this is the respose after debug

if you see I want to use "status" =\> "200" add it to the elastic search but as per current config file

it inserting this information "status" : "%{[http\_poller\_metadata][code]}", in elastic search.

---

<div class="post-metadata">

**Author:** ![Pankaj\_Jadhav](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pankaj_jadhav/32/65896_2.png) [@Pankaj\_Jadhav](https://discuss.elastic.co/u/Pankaj_Jadhav)\
**Post date:** [June 3, 2020, 10:53am UTC](https://discuss.elastic.co/t/how-to-send-http-poller-response-data-in-elastic-search/235519/2 "2020-06-03T10:53:35Z")

</div>

Is there any event or method which will trigger in logstash/filter after http\_poller response because am getting 2 object and it inserting value of 1st object but my 2nd object response is the http\_poller response i've to use the "status" from that object and add/update in elastic search

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 1, 2020, 10:53am UTC](https://discuss.elastic.co/t/how-to-send-http-poller-response-data-in-elastic-search/235519/3 "2020-07-01T10:53:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.