Error event executor terminated

Elastic Stack Logstash
1

Logstash was having pipeline error: event executor terminated. What would possibly be the problem??

Filebeat (v. 6.3.2) - filebeat.yml:

filebeat.inputs:
- type: log
  enabled: true
  paths:
   - c:\var\logs\oob-demo\safebanking-sb*.log
  tags: ["safebanking"]
  fields:
    host_ip: 192.168.1.50
    
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
  
setup.template.settings:
  index.number_of_shards: 3
  
setup.kibana:
  host: "192.168.1.79:5601"
  
output.logstash:
  hosts: ["192.168.1.79:5044"]
  username: "filebeat_internal"
  password: "XXXXXX"
  
logging.level: info
logging.to_files: true
logging.files:
  path: 'C:\Program Files\Filebeat\log'
  name: filebeat
  keepfiles: 7
  permissions: 0644

Logstash (v. 6.2.3) => filebeat plugin (v. 5.0.10) - config file:

input {
  if "safebanking" in [tags] {
    beats {
      port => 5044
      codec => "json"
    }
  } else {
    beats {
      port =>5044
    } 
  }
}
filter {
  if "safebanking" in [tags] {
    json {
      skip_on_invalid_json => true
      source => "message"
    }
  }
  mutate {
    remove_field => [ "host" ]
  }
}
output {
  if "safebanking" in [tags] {
    elasticsearch { 
        hosts => ["192.168.1.79:9200"] 
        user => "logstash_internal"
        password => "Not4u2know"
        index => "safebanking-%{+YYYY.MM.dd}"
    }
  }
  if "tde" in [tags] {
    elasticsearch {
      hosts => ["192.168.1.79:9200"]
      user => "logstash_internal"
      password => "Not4u2know"
      index => "tde-%{+YYYY.MM.dd}"
    }
  }
  stdout { codec => rubydebug }
}

logstash-plain.log:
[2018-08-15T13:54:40,846][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Beats port=>5044, codec=><LogStash::Codecs::JSON id=>"json_2f27aae1-6f00-419b-8a9c-8161e0f768da", enable_metric=>true, charset=>"UTF-8">, id=>"816e43d4c21299155ff1ac45e24216b52ee7ee7ddaa5a45f3bb6d9af54edea35", enable_metric=>true, host=>"0.0.0.0", ssl=>false, ssl_verify_mode=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>4>
Error: event executor terminated
Exception: Java::JavaUtilConcurrent::RejectedExecutionException
Stack: io.netty.util.concurrent.SingleThreadEventExecutor.reject(io/netty/util/concurrent/SingleThreadEventExecutor.java:821)
io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(io/netty/util/concurrent/SingleThreadEventExecutor.java:327)
io.netty.util.concurrent.SingleThreadEventExecutor.addTask(io/netty/util/concurrent/SingleThreadEventExecutor.java:320)
io.netty.util.concurrent.SingleThreadEventExecutor.execute(io/netty/util/concurrent/SingleThreadEventExecutor.java:746)
io.netty.channel.AbstractChannel$AbstractUnsafe.register(io/netty/channel/AbstractChannel.java:479)
io.netty.channel.SingleThreadEventLoop.register(io/netty/channel/SingleThreadEventLoop.java:80)
io.netty.channel.SingleThreadEventLoop.register(io/netty/channel/SingleThreadEventLoop.java:74)
io.netty.channel.MultithreadEventLoopGroup.register(io/netty/channel/MultithreadEventLoopGroup.java:86)
io.netty.bootstrap.AbstractBootstrap.initAndRegister(io/netty/bootstrap/AbstractBootstrap.java:331)
io.netty.bootstrap.AbstractBootstrap.doBind(io/netty/bootstrap/AbstractBootstrap.java:282)
io.netty.bootstrap.AbstractBootstrap.bind(io/netty/bootstrap/AbstractBootstrap.java:278)
io.netty.bootstrap.AbstractBootstrap.bind(io/netty/bootstrap/AbstractBootstrap.java:260)
org.logstash.beats.Server.listen(org/logstash/beats/Server.java:57)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:438)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:302)
usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_beats_minus_5_dot_0_dot_10_minus_java.lib.logstash.inputs.beats.invokeOther4:listen(usr/share/logstash/vendor/bundle/jruby/$2_dot_3_dot_0/gems/logstash_minus_input_minus_beats_minus_5_dot_0_dot_10_minus_java/lib/logstash/inputs//usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/lib/logstash/inputs/beats.rb:198)
usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_beats_minus_5_dot_0_dot_10_minus_java.lib.logstash.inputs.beats.run(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-beats-5.0.10-java/lib/logstash/inputs/beats.rb:198)
RUBY.inputworker(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:516)
RUBY.block in start_input(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:509)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)
java.lang.Thread.run(java/lang/Thread.java:748)

2

You cannot use a conditional in an input block. You are asking it to use a different input to receive an event, based on a field in the event. That would require it to know the contents of the event before it received it. I would expect you to be getting a "Address already in use" error because you have two beat inputs listening on the same port.

Also, it is unlikely that you want both a json filter and a json codec (you would only need that if the event were JSON that contained a nested JSON field called message). Something like:

{ "message": "{ \"foo\": 1}" }
3

[UPDATE] I put the channels in the same input and used filter in the pipeline to differentiate them. I did not use the json filter because it's not compatible with double-quotes in the values anyways. I manually split the json-formatted logs using kv filters and did otherwise for the non-json logs.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.