Hi,
I've upgraded my Logstash instances from 6.8 to 7.3. After the upgrade, I've noticed a startup time ~3x higher, going from ~3min to ~9min. I'm surprised because I red in the release notes that the startup time is expected to be much faster with the new Java engine.
To give you more details about what I observe, the startup behavior is actually different from 6 to 7.
In version 6, during startup, in the logs, I had nothing for ~3min and then in a few seconds, I had all these lines and Logstash was 100% ready and processing logs :
Starting UDP listener {:address=>"0.0.0.0:xxxxx"}
UDP listener started {:address=>"0.0.0.0:xxxxx", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
Starting UDP listener {:address=>"0.0.0.0:yyyyy"}
UDP listener started {:address=>"0.0.0.0:yyyyy", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
....
Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
Opening file {:path=>"/var/log/xxxxxx"}
....
But in version 7 it's different. In ~1min after the startup command, I get these logs :
Starting UDP listener {:address=>"0.0.0.0:xxxxx"}
UDP listener started {:address=>"0.0.0.0:xxxxx", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
Starting UDP listener {:address=>"0.0.0.0:yyyyy"}
UDP listener started {:address=>"0.0.0.0:yyyyy", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
....
Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
So at that time it looks like the pipeline is ready. But in fact not. There is no output at that time. The output starts coming ~8 minutes later. And during this time where the inputs are ready, the pipeline is supposed to run but there is no output. I've strace the process and it's actually generating all the Java CompileDataset in /tmp. In my case there are 400 files like that to generate. After the last file is generated, the output starts coming.
So it looks like the pipeline is ready in ~1min, but in fact it takes really 9min.
Do you know why it takes so long ? Did I miss a configuration option while upgrading to 7.x ?
In my pipeline I use many custom Ruby filters, both inline and using an external Ruby script files.
Could it explain this high Java generation time ?
Thank you for your help