I'm currently running Logstash with automated configuration reload so that I may automate the deployment of new configuration without triggering a restart.
I noticed that automated configuration reload is quite CPU intensive. I originally configured a 5 second delay, only to notice constant high CPU usage. Reducing it from 5 seconds to 5 minutes made it better.
By letting logstash run constantly in that mode, I also noticed that CPU consumption to reload the configuration seems to increase. For instance, a config reload used to spike the CPU for a few seconds, but after running for 4 weeks, the same config reload seems to cause a spike of 2 minutes. If I restart Logstash, those spikes get back to a few seconds.
To solve this, I can disable automatic configuration reload and trigger a kill -1 to force the reload.
However, would it be better to make Logstash check for file modifications before recreating the pipelines?
Odds are, the CPU cost of doing a few file size/date/content change checks will be a small fraction of the cost to load them, compile them and stop-start all the pipelines.