Hey all,
I've noticed the (apparently planned) behavior of logstash where it will restart if the configuration file isn't valid, even if the configuration file isn't changed. While the ideal situation is don't get the configuration wrong, there's still times where an unexpected/untested change happens and we're seeing constant restarts, and cause massive CPU load. Our environment is under config management and sometimes a bad change will cause the entire fleet of nodes to jump to 90% CPU until a fix is pushed. Yes, we're working on better testing/deploy processes etc to stop this from being a problem, but, in the interim (and as a safety measure) is there an option to either:
- Throttle the frequency of Logstash trying to restart from near-instant to say 5 minutes if there is bad configuration
- Get Logstash to not auto-restart if the config is bad UNTIL the config file has changed ("The definition of insanity is doing the same thing over and over again and expecting a different result")
- Force Logstash to run config.test_and_exit and not restarting if the new config doesn't parse
- Other suggestions?
Looked for any available parameters and the closest that could be found were config.reload.automatic and config.reload.interval - both of which don't seem to impact failed config tests.
Environment:
- Logstash 7.x
- Ubuntu 18.04
Cheers, CCM