Input S3 with debug mode (BIG POST)

[Juan_Andres_Ramirez]

Hello ,
For a couple days I have been working with logstash 6.2 with docker, using the official docker's image for try get S3 file with logs, I'll try describe all the process with lot details, because I can't do get logs with logstash.

1- The following config I have been using to format files, basically I'm using grok filter, previously check it, in the grok tester online (http://grokconstructor.appspot.com)

pipeline.yml

input {
      s3 {
        bucket => "e-api-logs"
        prefix => "api/test/*.txt"
        interval => 30
        access_key_id => "xxxxxxx"
        secret_access_key => "xxxxx"
      }
    }

output {
      if "-grokparsefailure" not in [tags] {
        elasticsearch {
          hosts => ["http://elastic-svc:9200"]
          index => "e-api-%{+YYYY.MM.dd}"
        }
        stdout { codec => rubydebug }
      }
    }

`

Is not necessary show the grok filter because I don't have errors with it.

I ran the docker with logstash in debug mode:

1. • Starting logstash:

     [INFO ][logstash.runner] Starting Logstash {"logstash.version"=>"6.2.2"}
     [2018-03-15T13:52:12,181][DEBUG][logstash.agent] Starting agent
     [2018-03-15T13:52:12,195][DEBUG][logstash.agent] Starting puma
     [2018-03-15T13:52:12,206][DEBUG][logstash.agent] Trying to start WebServer 
        {:port=>9600}
     [INFO ][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
     

2. • S3 plugin:

     [DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"s3", :type=>"input", :class=>LogStash::Inputs::S3}
     [2018-03-15T13:52:22,512][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"plain", :type=>"codec", :class=>LogStash::Codecs::Plain}
     [DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@enable_metric = true
     [2018-03-15T13:52:22,539][DEBUG][logstash.codecs.plain    ] config 
     LogStash::Codecs::Plain/@charset = "UTF-8"
     [2018-03-15T13:52:22,553][DEBUG][logstash.inputs.s3       ] config 
     LogStash::Inputs::S3/@bucket = "e-api-logs"
     [2018-03-15T13:52:22,553][DEBUG][logstash.inputs.s3       ] config 
     LogStash::Inputs::S3/@prefix = "api/test/*.txt"
     [2018-03-15T13:52:22,553][DEBUG][logstash.inputs.s3       ] config 
     LogStash::Inputs::S3/@interval = 30
     [2018-03-15T13:52:22,553][DEBUG][logstash.inputs.s3       ] config 
     LogStash::Inputs::S3/@access_key_id = "xxxxxxxxx"
     [2018-03-15T13:52:22,553][DEBUG][logstash.inputs.s3       ] config 
     LogStash::Inputs::S3/@secret_access_key = "xxxxxxxxxxx"
     [2018-03-15T13:52:22,553][DEBUG][logstash.inputs.s3       ] config LogStash::Inputs::S3/@id = "3bb5e1255"
     [2018-03-15T13:52:22,554][DEBUG][logstash.inputs.s3       ] config LogStash::Inputs::S3/@enable_metric = true
     [DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"grok", :type=>"filter", :class=>LogStash::Filters::Grok}
     [DEBUG][logstash.filters.grok    ] Grok compiled OK
     

3. • Connecting to S3

     [INFO ][logstash.inputs.s3       ] Registering s3 input {:bucket=>"ecdf-api-logs", :region=>"us-east-1"}
     [2018-03-15T13:52:32,047][INFO ][logstash.pipeline        ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2d382e65 run>"}
     [2018-03-15T13:52:32,136][INFO ][logstash.agent           ] Pipelines running {:count=>1, :pipelines=>["main"]} 
     

Everything looks good for now, but I have 2 problems, the first I don't have any output (remember it has rubydebug activated) and I have an error in debug mode:

[DEBUG][logstash.instrument.periodicpoller.cgroup] Error, cannot retrieve cgroups information {:exception=>"Errno::ENOENT", :message=>"No such file or directory - /sys/fs/cgroup/cpuacct/kubepods/besteffort/pod00ce1eec-2858-11e8-bc5c-0297d68e3126/5e404648a476ab9f4667f6d09bee669bc939bd8318a227ee6737d8a48c8f5cb8/cpuacct.usage"}
[2018-03-15T13:52:35,257][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-03-15T13:52:35,259][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-03-15T13:52:37,063][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x2d382e65 sleep>"}

It has repeat at the end of logs.

Any help I'll appreciate, thanks.