Logstash pipelines throwing NoMethodError exceptions with undefined method ever since I added an S3 Output plugin (6.4)

[shog1466112]

Hi there,

I recently added an S3 output plugin to my logstash pipelines (two of them). Ever since I've been getting these exceptions which I haven't been able to get to the bottom of.

The containers running logstash definitely have the correct IAM policy assigned with permission to write to the buckets as I'm getting logs output to S3 now too, however it seems that Logstash is crashing randomly with these exceptions.

exception=>"NoMethodError", :message=>"undefined method `call' for nil:NilClass"

Here is a more complete log from one of the container tasks that crashed: https://pastebin.com/6bx4A8bd

In terms of container config, here are my pipelines:

- pipeline.id: ingest_redis_clones
  path.config: "/usr/share/logstash/pipelines/ingest_redis_clones.conf"

- pipeline.id: ingest_gg_app_logs
  path.config: "/usr/share/logstash/pipelines/ingest_gg_app_logs.conf"

Here is the relevant output section for one of these pipelines. The S3 output is identical for both pipelines.

output {
  if [type] != "discard" {
    lumberjack {
      hosts => ["xxxxxxx"]
      port => 5006
      ssl_certificate => "/etc/ssl/xxxx.crt"
      codec => "json_lines"
    }

    s3 {
      bucket => "xxxx-xxx-output-bucketstore-prod"
      region => "eu-west-1"
      codec => "json_lines"
      canned_acl => "private"
      prefix => "%{type}"
      upload_queue_size => 4
    }

  } else {
    null {
      # output to null (discard)
    }
  }
}

Any ideas what might be causing these exceptions? I never saw them before making the change to add the additional output (S3). Also, I've tried this with both Logstash 6.3 and 6.4 OSS official docker images.

Thanks!

[shog1466112]

Any ideas on this one anyone?

My Fargate task containers still randomly fail on start up with this strange exception. But others start just fine. No changes to parameters, versions or anything. Just sometimes they initialize correctly and sometimes they don't. Luckily the container dies in latter scenario and Fargate initializes a new container, so generally after a few minutes I'll have a full set of Logstash tasks running to do their work (redis input)...