Is there a way to alter the value of live API data polled using http_poller input plugin?

[APJ]

Hi all,

I have a logstash config file that pulls in live data using the http_poller plugin. Here is my logstash.conf

input {
  http_poller {
    urls => {
        weather => {
            url => "http://api.openweathermap.org/data/2.5/weather?id=5490223&appid="MY_APP_ID"&units=metric"
            headers => {
              Accept => "application/json"
            }
        }
    }
    schedule => { cron => "* * * * * *" }
    codec => json
  }
}
filter {
  mutate {
    remove_field => ["@version" ,"command" ,"host" ,"cod" ,"id" ,"base" ,"coord" ,"sys" ,"dt"]
  }
  split { field => "weather" }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "weather"
  }
  stdout {
    codec => rubydebug
  }
}

I want to modify the data retrieved from openweathermap every 30 seconds before sending it into elasticsearch. Is there a way to do it?

I would really appreciate if someone could help me out. Thanks!

[NerdSec (Nachiket)]

@APJ

I want to modify the data retrieved from openweathermap every 30 seconds before sending it into elasticsearch. Is there a way to do it?

What do you mean by modify? Could you give an example?

[APJ]

Hi NerdSec,

The http://api.openweathermap.org/data/2.5/weather?id=5490223&appid="MY_APP_ID"&units=metric API gives me a lot of data (fields), like temperature, humdity, pressure, etc.

For instance, I want to alter the actual temperature value polled in by logstash, and then send that value to elasticsearch.

Let's assume the main.temp is 20ºF. Now, I want logstash to make it 30ºF and send that data to elasticsearch

So, I'd like to know if there is a way to do this. Please help me out here.

[NerdSec (Nachiket)]

Hi,

You can do arithmetic operations using the ruby filter. Here are some links.

Event API | Logstash Reference [8.11] | Elastic

Doing mathematical ops using logstash filters

if 'time_duration' < '500' { mutate { add_tag => [ "FAST" ] } Two problems: Fields are referred via the square bracket notation, i.e. [time_duration] in this case. You need fields to be integers (or doubles) for numerical comparisons to work. See below. I am using haproxy config, which I don't think I need to print here. But for reference, the bytes_read and the time_duration are INT types. No, they're strings. That's obvious from the stdout output. The fact that a grok expression use…

Math operations with ruby

I want to multiply or divide two fields and create a new field based on that. I've got this ruby code: ruby { code => "event['hit_rate'] = event['MAIN_client_req'] * event['MAIN_cache_hit']" } When I try to use it I'm getting: Ruby exception occurred: undefined method `*' for nil:NilClass {:level=>:error} Ruby exception occurred: NilClass can't be coerced into Fixnum {:level=>:error} This is a part of my logstash config: grok { match => { "message" => "(?<par…

Math functions in Logstash

Regards,
NerdSec

[APJ]

Thanks a lot NerdSec!

My goal is to perform addition operation on main.temp value (main.temp = main.temp + 10) every 30 seconds. Elasticsearch is something new to me. So I would be grateful if you could let me know what exactly needs to be added or changed in the logstash.conf.

Regards,
APJ

[NerdSec (Nachiket)]

I am not sure how to do it for every 30 sec. As far as I am aware, the said filter will run for every event, irrespective of time.