S3 input stack trace and multi-pipeline config question

Elastic Stack Logstash
1

Hello

Had a stack trace appear when using the following configs, there are a few more configs I couldn't fit in this post, but are very similar to the one below. I found including more than 1 config in a single pipeline resulted in events ending up in the wrong output, so I defined a pipeline per AWS flow log bucket.

Does this look like a sane way of managing multiple s3 inputs going to their own syslog output? Pipelines seem well suited otherwise I'd be doing lots of tagging and conditionals in one config. Stack trace at the bottom. Grateful for any thoughts.

pipelines.yml:

- pipeline.id: 25da1c4c-pipline
   path.config: "/etc/logstash/conf.d/aws-flowlogs-eu-central-1-25da1c4c.conf"
 - pipeline.id: 06f2f234a6d3b142a-pipline
   path.config: "/etc/logstash/conf.d/aws-flowlogs-eu-central-1-06f2f234a6d3b142a.conf"
 - pipeline.id: 2b47e44e-pipline
   path.config: "/etc/logstash/conf.d/aws-flowlogs-eu-west-1-2b47e44e.conf"
 - pipeline.id: cd8909a9-pipline
   path.config: "/etc/logstash/conf.d/aws-flowlogs-ap-southeast-1-cd8909a9.conf"

conf.d/*

input {
    s3 {
	bucket => "flowlogs-ireland-2b47e44e"
	sincedb_path => "/var/lib/logstash/sincedb-flowlogs-ireland-2b47e44e"
	type => "s3"
	region => "eu-west-1"
	exclude_pattern => "(?:(2017\/[0-1])|2018\/0)"
	interval => 300
    }
}

output {
  syslog {
    host => "mylogserver"
    protocol => "ssl-tcp"
    port => 6514
    facility => "local0"
    severity => "informational"
    appname => "aws-flowlogs-eu-west-1-2b47e44e"
    ssl_cacert => "/etc/syslog-ng/ca.d/cacert.pem"
    ssl_cert => "/etc/pki/tls/certs/hostcert.pem"
    ssl_key => "/etc/pki/tls/certs/hostkey.pem"
    sourcehost => "mylogforwarder"
  }
}


logstash[29981]: [8,955][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
logstash[29981]: Pipeline_id:2b47e44e-pipline
logstash[29981]: Plugin: <LogStash::Inputs::S3 bucket=>"flowlogs-ireland-2b47e44e", exclude_pattern=>"(?:(2017\\/[0-1])|2018\\/0)", interval=>300, id=
>"c1fbe0cf05e8080b6b81f1190cc700636aa18c7d9bfe8a75b061ab4a6e8a0e72", sincedb_path=>"/var/lib/logstash/sincedb-flowlogs-ireland-2b47e44e", type=>"s3", region=>"eu-west-1", enable_metric=>true, codec=>
<LogStash::Codecs::Plain id=>"plain_93dca249-a400-4c97-9968-b68e8693e255", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", delete=>false, temporary_directory=>"/tmp/logstash", include_obje
ct_properties=>false>
logstash[29981]: Error:
logstash[29981]: Exception: Java::JavaLang::NullPointerException
logstash[29981]: Stack: org.jruby.RubyString.getStringForPattern(RubyString.java:3741)
logstash[29981]: org.jruby.RubyString.asRegexpArg(RubyString.java:2405)
logstash[29981]: org.jruby.RubyString.subBangNoIter(RubyString.java:2445)
logstash[29981]: org.jruby.RubyString.sub_bang(RubyString.java:2398)
logstash[29981]: org.jruby.RubyString$INVOKER$i$sub_bang.call(RubyString$INVOKER$i$sub_bang.gen)
logstash[29981]: org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrTwoOrNBlock.call(JavaMethod.java:402)
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.