Dynamic variable for http poller input plugin

Elastic Stack Logstash
1

I want to download and parse a list of csv with yearly weather data with logstash.
"YEAR" would be my variable and I need to increase from 1998 to 2019.
I tried to use this simple conf file but not even works.

input {
  http_poller {
    urls => {
      test1 => "http://weather.uwaterloo.ca/download/Daily_summary_${YEAR:1998}.csv"
     }
    request_timeout => 60
    schedule => { cron => "* * * * * UTC"}
    codec => "plain"
    metadata_target => "http_poller_metadata"
  }
}

filter{

}

output {
  stdout {
    codec => rubydebug
  }
  file {
     path => "/var/log/logstash/wheater-%{YEAR}.log"
  }
}
2

Hi there,

what do you mean by "not even works"? What does it do?

Anyway, AFAIK, the only way to pass variables to Logstash is through environment variables.
If you do not have a env variable YEAR specified, it should take the default one you passed (1998).

Obviously it won't find anything in your output section, so I'd expect it to write something in the /var/log/logstash/wheater-%{YEAR}.log literal path.

But what does it write in your standard output?

3

I changed codec in plain. It was wrong.

Correct .

With codec plain logstash save file and data from Daily_summary_1998.csv file

4

Ok so, it does what it is expected to be done. I don't really get why you're using Logstash to do something that can easily be done with a script.

I'd use a script (where you can make dynamic variables) to save those csv files and then I'd parse them with Logstash if needed.

It definitely is a better solution rather than storing I don't know how many env variables or launch Logstash so many times.

5

Yes. I thought there was a simple solution to do this but it seems that it's not.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.