Logstash javapipeline error=> (OpenTimeout) execution expired

Elastic Stack Logstash
1

Hi Teams.

The problem is in the pipeline.
I noticed that data periodically stops flowing for one index. I checked and after a certain time the port stops "listening". At the input, TCP is listening, I saw this message in the logs:

[ERROR][logstash.javapipeline    ][expressway-pipe] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"expressway-pipe", :error=>"(OpenTimeout) execution expired", :exception=>Java::OrgJrubyExceptions::RuntimeError, :backtrace=>["org.jruby.RubyRegexp.match?(org/jruby/RubyRegexp.java:1132)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304)"], :thread=>"#<Thread:0x4c60c64f /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}

Logstash version 8.13.3

At first the data is coming, but then problems start? Any ideas?
Thanks.

2

Hi, update.

Noticed a pipeline problem again, but with a different port. The error is the same. Any ideas? I'd be glad for any ideas.
Logstash version = 8.13.3 elastic version = 8.15.0. Could this cause some compatibility issues? But according to the matrix there shouldn't be any or am I missing something?

3

Hi. I started to notice some more problems on one more port.
I updated logstash to version 8.15 but still have problems. I'm losing logs, any ideas?

[ERROR][logstash.javapipeline    ][gw-pipe] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"gw-pipe", :error=>"(OpenTimeout) execution expired", :exception=>Java::OrgJrubyExceptions::RuntimeError, :backtrace=>["org.jruby.RubyRegexp.match?(org/jruby/RubyRegexp.java:1138)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304)"], :thread=>"#<Thread:0x125776d0 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}
[WARN ][logstash.javapipeline    ][gw-pipe] Waiting for input plugin to close {:pipeline_id=>"gw-pipe", :thread=>"#<Thread:0x125776d0 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
4

Team, does anyone have any ideas? Why the port closes after a certain time and I have to make a comment in the pipeline to get it to start again.

5

Have you set the parameter: tcp_keep_alive?

Instruct the socket to use TCP keep alive. If it’s true then the underlying socket will use the OS defaults settings for keep alive. If it’s false it doesn’t configure any keep alive setting for the underlying socket. * Default value is false

Be aware this might be related to OS also since you are the TCP plugin and have the error: (OpenTimeout) execution expired

Try to set the param to tcp_keep_alive => true and review OS TCP settings ( sysctl -a).

6

Hi Rios !
thanks for the recommendations. I'll check it for TCP input. But I have tcp and udp, the problem is similar. The most difficult thing is that I observe this on several configs, I haven't noticed such problems on other spacebars (I have two logstash nodes and a balancer).

The strangest thing is that I hadn’t noticed any problems before - the port itself stopped listening ...

7

Something must be different:

  • source of data and who/how they send
  • OS settings
  • LS settings
  • ELK versions
8

maybe you're right, but what changed was that I updated the ELK + logstash cluster. data was just coming from these sources and there were no comments before. the configuration didn't change...

Now the most important thing is to understand how to restore stability.

9

You have to investigate, I have provided ideas what could be. I'm sure someone else can have other suggestions as well.

Maybe a ruby package on which is based LS module has come issues. Hard to know.
You can set log.level: "trace" and investigate deeper.