Pipeline starts before the compilation is completed with Java execution

Hi, everyone.

I've been trying out the Java execution engine in Logstash 6.6.0 since it's now considered GA.

We have several Logstash instances running the same pipeline with Persistent Queue enabled. Sometimes we need to fix an issue with the pipeline or add something new to it. We've been using an Ansible role to do that: it sequentially updates all instances, waiting for one to become available before moving on to the next one.

This allows us to perform rolling updates whenever we want and under any load: only one Logstash instance may be unavailable at a time and if anything goes wrong the update fails immediately and doesn't propagate throughout the whole system.

Now, with the Ruby execution engine this was trivial to implement via a chain of handlers: they restart or reload the instance and then just wait for inputs to go up. This doesn't work with Java execution because the pipeline starts before the compilation of filters and conditionals is completed. So it goes like this: we restart/reload Logstash, the pipeline (and all inputs) launches and starts ingesting events into the Persistent Queue. These events accumulate in the queue because the compilation process needs another 2-3 minutes. Eventually, the compilation is completed successfully and Logstash starts processing the (quite large at this point) Persistent Queue.

This doesn't work for us, because we want to know the exact moment in time when Logstash is ready to process events so we can perform a safe rolling update of our pipeline. Is there any way we can detect the fact that the compilation process is still not finished without disabling the PQ or enabling debug logging?

@KBuev, I'd like to verify this difference in behavior between the Ruby and Java execution engines. Just for the sake of clarification, though, you're saying it takes 2-3 minutes for the Java execution to compile your pipeline? I presume it is a pretty large one, then?

Just for the sake of clarification, though, you're saying it takes 2-3 minutes for the Java execution to compile your pipeline?

Yep, it takes around 3-4 minutes with debugging enabled (that's how we determined that the compilation is what's causing the delay) and 2-3 minutes with log.level:info

Here are some details about our setup:

Logstash:

  • 4 VMs
  • Each one has 8 cores and 24GiB of RAM
  • Each Logstash instance has a heap of 16GiB and 16 pipeline workers

Pipeline:

  • 4617 lines
  • 26 files
  • 140KiB

Filter plugins:

  • 1 split
  • 2 kv
  • 3 json
  • 3 xml
  • 5 useragent
  • 10 drop
  • 12 fingerprint
  • 12 geoip
  • 16 ruby
  • 41 date
  • 98 grok
  • 195 mutate

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.