Hi, everyone.
I've been trying out the Java execution engine in Logstash 6.6.0 since it's now considered GA.
We have several Logstash instances running the same pipeline with Persistent Queue enabled. Sometimes we need to fix an issue with the pipeline or add something new to it. We've been using an Ansible role to do that: it sequentially updates all instances, waiting for one to become available before moving on to the next one.
This allows us to perform rolling updates whenever we want and under any load: only one Logstash instance may be unavailable at a time and if anything goes wrong the update fails immediately and doesn't propagate throughout the whole system.
Now, with the Ruby execution engine this was trivial to implement via a chain of handlers: they restart or reload the instance and then just wait for inputs to go up. This doesn't work with Java execution because the pipeline starts before the compilation of filters and conditionals is completed. So it goes like this: we restart/reload Logstash, the pipeline (and all inputs) launches and starts ingesting events into the Persistent Queue. These events accumulate in the queue because the compilation process needs another 2-3 minutes. Eventually, the compilation is completed successfully and Logstash starts processing the (quite large at this point) Persistent Queue.
This doesn't work for us, because we want to know the exact moment in time when Logstash is ready to process events so we can perform a safe rolling update of our pipeline. Is there any way we can detect the fact that the compilation process is still not finished without disabling the PQ or enabling debug logging?