Get json from Http poller , store it in local variable and use it in elastic search output

[Abolurah (Akiva)]

Hi All
I'm trying to configure the Logstash, that it will poll credentials from Http poller plugin store it is some local variable, get data from local elastic search (multiple Inputs), and will send it to remote AWS Elasticsearch with credentials I got from Http-poller JSON.
1- what is the best way to do that?
2- cant find an example of using local variables in a config file (except this one
ruby {
init => "@counter = 0"
code => "event.set('message_count', @counter)"
})
Your help will be appreciated.

My config example

 input { 	
	elasticsearch {
		hosts => "${LOCAL_IP}:9200" 	
		index => "errorevents" 
		type => "errorevent"
		docinfo => true		
		query => '{"query": {"match_all": {}}}'
		schedule => "* * * * *"	
	}	
	
input {
  http_poller {
    urls => {
      test2 => {
        url => "${LOCAL_IP}:8080/iot/resources"
     headers => {accept => "application/json"}
			 }
    }
    request_timeout => 10
    schedule => { cron => "* * * * * utc"}
			codec => "json"
			metadata_target => "http_poller_metadata" }
	}

output {

stdout{id => "stdout_plugin"}


amazon_es {
	hosts => ["https://fake.amazonaws.com"] #Here I want to use local variable or data from json I got, instead of hard coded values 
	region => "us-east-f8"  #Here I want to use local variable or data from json I got, instead of hard coded values
	aws_access_key_id => 'FAKELNKLNDLNDJKNDJKDJKDNDJK'  #Here I want to use local variable or data from json I got, instead of hard coded values
	aws_secret_access_key => 'FAKElksnmclknsfkvndjkfbvdjkfbvjdkf' #Here I want to use local variable or data from json I got, instead of hard coded values
	index => "error-${DEVICE_ID}"
	}

}

[Badger]

There is no way of ensuring that data from the elasticsearch input will be processed after the data from the http_poller input, which makes this problematic.

The output can use sprintf references to fields on the event (including fields under [@metadata]).

If you are running the elasticsearch input and http_poller input on the same schedule then you might be able to do this by removing the elasticsearch input and using an elasticsearch filter instead to run the query. That way the event will have access to both the http_poller data and the elasticsearch data.

[Abolurah (Akiva)]

Thanks for reply.
What if I will use elasticsearch as input , but inside filter I will make a rest call that will bring me the credentials back , and I will use them in output .
Do you think is it better resolution ?
I will be thankful for such working example.
Thanks.

[Abolurah (Akiva)]

Hi @magnusbaecki, @Badger
What do you think about this?

[Abolurah (Akiva)]

Add to input

	  http_poller {
		  type => "httppoller"	
			urls => {
			  test2 => {
				# Supports all options supported by ruby's Manticore HTTP client
				method => get

				url => "${LOCAL_IP}:8080/iot/resources"
				headers => {
				  Accept => "application/json"
				}
			 }
			}
			request_timeout => 10

			# Supports "cron", "every", "at" and "in" schedules by rufus scheduler
			schedule => { cron => "* * * * * UTC"}
			codec => "json"
			# A hash of request metadata info (timing, response headers, etc.) will be sent here
			metadata_target => "http_poller_metadata"
		  }

Add filter

filter {

		json{source => "message"}
		
		if [type] == "httppoller" {
			
		
			
			ruby { code => '@@aws_access_key_id = event.get("aws_access_key_id")' }	
			
			drop{}
			
			
					
		}
		
		ruby{
			code => '	
						
						
						if defined?(@@aws_access_key_id)
                                                            event.set("[@metadata][my_aws_access_key_id]", @@aws_access_key_id)
															
                                            else
                                                            event.set("[@metadata][my_aws_access_key_id]", "not-defined")
                                            end

						
					'
			}
		
		#Drop the event if credentials not defined			
		if [@metadata][my_aws_access_key_id] == "not-defined"
                            { drop{} }


}