# Get json from Http poller , store it in local variable and use it in elastic search output
**URL:** https://discuss.elastic.co/t/get-json-from-http-poller-store-it-in-local-variable-and-use-it-in-elastic-search-output/239610
**Category:** Logstash
**Created:** [July 2, 2020, 10:34am UTC](https://discuss.elastic.co/t/get-json-from-http-poller-store-it-in-local-variable-and-use-it-in-elastic-search-output/239610 "2020-07-02T10:34:11Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Abolurah](https://discuss.elastic.co/u/Abolurah)
#### Post date: [July 2, 2020, 10:34am UTC](https://discuss.elastic.co/t/get-json-from-http-poller-store-it-in-local-variable-and-use-it-in-elastic-search-output/239610/1 "2020-07-02T10:34:11Z")
Hi All
I'm trying to configure the Logstash, that it will poll credentials from Http poller plugin store it is some local variable, get data from local elastic search (multiple Inputs), and will send it to remote AWS Elasticsearch with credentials I got from Http-poller JSON.
1- what is the best way to do that?
2- cant find an example of using local variables in a config file (except this one
ruby {
init =\> "@counter = 0"
code =\> "event.set('message\_count', @counter)"
})
Your help will be appreciated.
My config example
```
input {
elasticsearch {
hosts => "${LOCAL_IP}:9200"
index => "errorevents"
type => "errorevent"
docinfo => true
query => '{"query": {"match_all": {}}}'
schedule => "* * * * *"
}
input {
http_poller {
urls => {
test2 => {
url => "${LOCAL_IP}:8080/iot/resources"
headers => {accept => "application/json"}
}
}
request_timeout => 10
schedule => { cron => "* * * * * utc"}
codec => "json"
metadata_target => "http_poller_metadata" }
}
```
output {
```
stdout{id => "stdout_plugin"}
amazon_es {
hosts => ["https://fake.amazonaws.com"] #Here I want to use local variable or data from json I got, instead of hard coded values
region => "us-east-f8" #Here I want to use local variable or data from json I got, instead of hard coded values
aws_access_key_id => 'FAKELNKLNDLNDJKNDJKDJKDNDJK' #Here I want to use local variable or data from json I got, instead of hard coded values
aws_secret_access_key => 'FAKElksnmclknsfkvndjkfbvdjkfbvjdkf' #Here I want to use local variable or data from json I got, instead of hard coded values
index => "error-${DEVICE_ID}"
}
```
}
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 2, 2020, 3:38pm UTC](https://discuss.elastic.co/t/get-json-from-http-poller-store-it-in-local-variable-and-use-it-in-elastic-search-output/239610/2 "2020-07-02T15:38:27Z")
There is no way of ensuring that data from the elasticsearch input will be processed after the data from the http\_poller input, which makes this problematic.
The output can use [sprintf](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#sprintf) references to fields on the event (including fields under [@metadata]).
If you are running the elasticsearch input and http\_poller input on the same schedule then you might be able to do this by removing the elasticsearch input and using an elasticsearch filter instead to run the query. That way the event will have access to both the http\_poller data and the elasticsearch data.
---
### Author:  [@Abolurah](https://discuss.elastic.co/u/Abolurah)
#### Post date: [July 5, 2020, 7:50am UTC](https://discuss.elastic.co/t/get-json-from-http-poller-store-it-in-local-variable-and-use-it-in-elastic-search-output/239610/3 "2020-07-05T07:50:48Z")
Thanks for reply.
What if I will use elasticsearch as input , but inside filter I will make a rest call that will bring me the credentials back , and I will use them in output .
Do you think is it better resolution ?
I will be thankful for such working example.
Thanks.
---
### Author:  [@Abolurah](https://discuss.elastic.co/u/Abolurah)
#### Post date: [July 6, 2020, 9:45am UTC](https://discuss.elastic.co/t/get-json-from-http-poller-store-it-in-local-variable-and-use-it-in-elastic-search-output/239610/4 "2020-07-06T09:45:06Z")
Hi @magnusbaecki, @Badger
What do you think about this?
---
### Author:  [@Abolurah](https://discuss.elastic.co/u/Abolurah)
#### Post date: [July 14, 2020, 9:56am UTC](https://discuss.elastic.co/t/get-json-from-http-poller-store-it-in-local-variable-and-use-it-in-elastic-search-output/239610/5 "2020-07-14T09:56:18Z")
# Add to input
```
http_poller {
type => "httppoller"
urls => {
test2 => {
# Supports all options supported by ruby's Manticore HTTP client
method => get
url => "${LOCAL_IP}:8080/iot/resources"
headers => {
Accept => "application/json"
}
}
}
request_timeout => 10
# Supports "cron", "every", "at" and "in" schedules by rufus scheduler
schedule => { cron => "* * * * * UTC"}
codec => "json"
# A hash of request metadata info (timing, response headers, etc.) will be sent here
metadata_target => "http_poller_metadata"
}
```
# Add filter
filter {
```
json{source => "message"}
if [type] == "httppoller" {
ruby { code => '@@aws_access_key_id = event.get("aws_access_key_id")' }
drop{}
}
ruby{
code => '
if defined?(@@aws_access_key_id)
event.set("[@metadata][my_aws_access_key_id]", @@aws_access_key_id)
else
event.set("[@metadata][my_aws_access_key_id]", "not-defined")
end
'
}
#Drop the event if credentials not defined
if [@metadata][my_aws_access_key_id] == "not-defined"
{ drop{} }
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 11, 2020, 9:56am UTC](https://discuss.elastic.co/t/get-json-from-http-poller-store-it-in-local-variable-and-use-it-in-elastic-search-output/239610/6 "2020-08-11T09:56:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.