Flexible interval of exec input

[BennyInc (Benny)]

We use an exec input in a Logstash configuration to poll the HA state of our database. Currently we set this run at an interval of 60, as the data isn't expected to change much during normal operation, and we don't want to spam our elastic with boring data.

However in maintenance scenarios (doing manual failovers) I would like to get the state more often, i.e. more fine-granular, with a reduced interval of 10 seconds.

Right now the only way I could do this would be by editing the config (change the interval) and then either restart Logstash, or have Logstash run with automatic config reload enabled.

However I would prefer to be able to set this dynamically, e.g. by putting a temp file in place that, if it exists, instructs Logstash to use a shorter interval.
First I had thought about using a file input which reads a file when it exists, then creates an event, which I would then use in a conditional to switch between two exec inputs with different intervals. I don't think that would work however, since inputs create events of their own, and cannot rely on other events.

Maybe someone here can think up a better hack to achieve what I want to do?

[BennyInc (Benny)]

Rubberducking to the rescue
I just found a solution that works for me. It has the undesired effect of calling the executable every 10s all the time, but at least it won't spam my elastic index. A simplified version using ls, so anyone can run it:

input {
  exec {
    command => "ls"
	interval => 10
  }
}

filter {
  ruby {
	code => 'unless File.exists?("C:/logstash_test/highfreq") then event.set("throttle", 0) end'
  }
  if [throttle] == 0 {
    throttle {
      key => "host"
      before_count => -1
	  after_count => 1
	  max_age => 7200
	  period => 60
	  add_tag => "throttled"
    }
    if "throttled" in [tags] {
      drop {}
    }
    mutate {
      remove_field => "throttle"
    }
  }
}

output {
  stdout {
    codec => "rubydebug"
  }
}

So I can touch a file of my chosing to unthrottle the events, while under normal circumstances having them throttled to one event per minute.

I could probably even further enhance it to always throttle to one event per minute (so I can monitor the "last" current state in Kibana) while unthrottling one single event if my HA state changed.

[BennyInc (Benny)]

So, the even better version now:

input {
  exec {
    command => "ls"
	interval => 5
  }
}

filter {
  throttle {
    key => "%{message}"
	before_count => -1
	after_count => 1
	period => 60
	add_tag => "throttled"
  }
}

output {
  stdout {
    codec => "rubydebug"
  }
}

In this sample, it will print the ls output every minute, unless a file is added or removed (i.e. the message changes), which will prompt an event immediately, once.