Logstash cloudwatch_logs input plugin has sincedb_path issues

Elastic Stack Logstash
1

According to lukewaite: Cloudwatch_logs input plugin the default sincedb_path for cloudwatch_logs is $HOME/.sincedb*". However, when I use this, I get some logs filtered, and I can see them being processed using the rubydebug, but then I get an error that keeps repeating, and no more logs are read from AWS Cloudwatch after.

Error:

[2019-06-26T20:47:03,411][ERROR][logstash.javapipeline    ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::CloudWatch_Logs start_position=>"beginning", log_group=>["/aws/lambda/", "/code/", "/aws/batch/", "/aws-glue/crawlers"], interval=>5, id=>"e6c72228bd2ed838a1dfab46e284edf6defac1cbf394491197fac15921ab0b6a", region=>"us-west-2", type=>"Cloudwatch", sincedb_path=>"$HOME/.sincedb*", log_group_prefix=>true, enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_18a099a3-d8d3-4552-a429-f51f299088c1", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash">
  Error: No such file or directory - $HOME/.sincedb*
  Exception: Errno::ENOENT
  Stack: org/jruby/RubyIO.java:1236:in `sysopen'
org/jruby/RubyIO.java:3796:in `write'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-cloudwatch_logs-1.0.3/lib/logstash/inputs/cloudwatch_logs.rb:250:in `_sincedb_write'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-cloudwatch_logs-1.0.3/lib/logstash/inputs/cloudwatch_logs.rb:198:in `block in process_group'
org/jruby/RubyKernel.java:1425:in `loop'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-cloudwatch_logs-1.0.3/lib/logstash/inputs/cloudwatch_logs.rb:182:in `process_group'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-cloudwatch_logs-1.0.3/lib/logstash/inputs/cloudwatch_logs.rb:123:in `block in run'
org/jruby/RubyArray.java:1792:in `each'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-cloudwatch_logs-1.0.3/lib/logstash/inputs/cloudwatch_logs.rb:121:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:297:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:290:in `block in start_input'

Logstash configuration:

input{
  cloudwatch_logs {
    log_group_prefix => true
    log_group => ["/aws/lambda/", "/code/", "/aws/batch/", "/aws-glue/crawlers"]
    region => "${AWS_REGION}"
    type => "Cloudwatch"
    interval => 5
    start_position => "beginning"
    sincedb_path => "$HOME/.sincedb*"
  }
}

filter{
  if [type] == "Cloudwatch" {
    if [event] != "" {
      mutate{
        add_field => {
          "[@metadata][tags]" => ["Cloudwatch"]
          "key" => "%{[Records][object][key]}"
        }
      }

      date {
        match => ["log-datestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
        target =>  "@timestamp"
        timezone => "UTC"
      }       
      date {
        match => ["log-datestamp", "YY-MM-dd HH:mm:ss,SSS"]
        target =>  "@timestamp"
        timezone => "UTC"
      }    
      date {
        match => ["log-datestamp", "ISO8601"]
        target =>  "@timestamp"
        timezone => "UTC"
      }    
      date {
        match => ["log-epoch", "UNIX"]
        target =>  "@timestamp"
        timezone => "UTC"
      }    
      date {
        match => ["log-epoch", "UNIX_MS"]
        target =>  "@timestamp"
        timezone => "UTC"
      }
    }
  }
}

output{
  if [type] == "Cloudwatch" {
    elasticsearch{
      hosts => ["${ES_HOST}"]
      user => "${USERNAME}"
      password => "${PASSWORD}"
      index => "${CW_INDEX}"
      document_id => "_cloudwatch"
    }
  }

  stdout { 
    codec => rubydebug {
      metadata => true
    }
  }
}
2

That will not work. If you want the default value then do not specify a value, the code will then generate a filename and create it under path.data.

3

The default is dev/null. The problem is, next time I run my program again, it keeps reading the logs that have been read, so I end up with duplicates.

Also, just to clarify, I am not using his code, but lukewaite has similar input plugin as I do.

4

That's not true if you are running the code I linked to.

5

@Badger, all I have in my directory is this:

kourosh:~/Desktop/dashboard_project$ ls
  Dockerfile  pipeline
kourosh:~/Desktop/dashboard_project$ ls pipeline/
  logstash.conf

I am not using lukewaite code at all. Do you think that is why it might be the problem?