Running Logstash in a Docker container version logstash:5.4.0 and I get the following log message after few seconds and after its sent few of the logs from s3 to ElasticSearch.
Logstash appears to be frozen, no updates sent to ElasticSearch, but it does not quit either.
Any suggestions on how to solve this problem ?
08:13:08.466 [[main]<s3] DEBUG logstash.inputs.s3 - S3 input: Download remote file {:remote_key=>"AWSLogs/111111/CloudTrail/us-east-1/2017/12/04/111111_CloudTrail_us-east-1_20171204T0000Z_OZxQsCeSQkZ23nhz.json.gz", :local_filename=>"/tmp/logstash/111111_CloudTrail_us-east-1_20171204T0000Z_OZxQsCeSQkZ23nhz.json.gz"}
08:13:09.317 [Ruby-0-Thread-13: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:559] DEBUG logstash.pipeline - Pushing flush onto pipeline
08:13:09.446 [[main]<s3] DEBUG logstash.inputs.s3 - Processing file {:filename=>"/tmp/logstash/111111_CloudTrail_us-east-1_20171204T0000Z_OZxQsCeSQkZ23nhz.json.gz"}
08:13:10.325 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - Reading config file {:config_file=>"/opt/logstash.conf"}
08:13:10.325 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - no configuration change for pipeline {:pipeline=>"main"}
08:13:12.322 [pool-2-thread-1] DEBUG logstash.instrument.periodicpoller.cgroup - Error, cannot retrieve cgroups information {:exception=>"Errno::ENOENT", :message=>"No such file or directory - /sys/fs/cgroup/cpuacct/docker/412f3811b237df6ac54cbcf3de325e68830f4e7894ff99e412963c7e691ed31d/cpuacct.usage"}
08:13:13.325 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - Reading config file {:config_file=>"/opt/logstash.conf"}
08:13:13.326 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - no configuration change for pipeline {:pipeline=>"main"}
08:13:14.320 [Ruby-0-Thread-13: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:559] DEBUG logstash.pipeline - Pushing flush onto pipeline
08:13:16.326 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - Reading config file {:config_file=>"/opt/logstash.conf"}
08:13:16.326 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - no configuration change for pipeline {:pipeline=>"main"}
08:13:17.335 [pool-2-thread-3] DEBUG logstash.instrument.periodicpoller.cgroup - Error, cannot retrieve cgroups information {:exception=>"Errno::ENOENT", :message=>"No such file or directory - /sys/fs/cgroup/cpuacct/docker/412f3811b237df6ac54cbcf3de325e68830f4e7894ff99e412963c7e691ed31d/cpuacct.usage"}
08:13:19.320 [Ruby-0-Thread-13: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:559] DEBUG logstash.pipeline - Pushing flush onto pipeline
08:13:19.329 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - Reading config file {:config_file=>"/opt/logstash.conf"}
08:13:19.329 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - no configuration change for pipeline {:pipeline=>"main"}
08:13:22.331 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - Reading config file {:config_file=>"/opt/logstash.conf"}
08:13:22.332 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - no configuration change for pipeline {:pipeline=>"main"}
08:13:22.344 [pool-2-thread-1] DEBUG logstash.instrument.periodicpoller.cgroup - Error, cannot retrieve cgroups information {:exception=>"Errno::ENOENT", :message=>"No such file or directory - /sys/fs/cgroup/cpuacct/docker/412f3811b237df6ac54cbcf3de325e68830f4e7894ff99e412963c7e691ed31d/cpuacct.usage"}
08:13:24.322 [Ruby-0-Thread-13: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:559] DEBUG logstash.pipeline - Pushing flush onto pipeline
08:13:25.325 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - Reading config file {:config_file=>"/opt/logstash.conf"}
08:13:25.325 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - no configuration change for pipeline {:pipeline=>"main"}
08:13:27.362 [pool-2-thread-4] DEBUG logstash.instrument.periodicpoller.cgroup - Error, cannot retrieve cgroups information {:exception=>"Errno::ENOENT", :message=>"No such file or directory - /sys/fs/cgroup/cpuacct/docker/412f3811b237df6ac54cbcf3de325e68830f4e7894ff99e412963c7e691ed31d/cpuacct.usage"}
08:13:28.324 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - Reading config file {:config_file=>"/opt/logstash.conf"}
08:13:28.324 [Ruby-0-Thread-3: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/task.rb:22] DEBUG logstash.agent - no configuration change for pipeline {:pipeline=>"main"}
Dockerfile:
FROM logstash:5.4.0
RUN logstash-plugin install logstash-codec-cloudtrail
RUN logstash-plugin install logstash-input-s3
CMD ["mkdir", "-p", "/opt"]
COPY logstash.conf /opt/
CMD ["-f", "/opt/logstash.conf", "--config.reload.automatic", "--debug"]
logstash conf file:
input {
s3 {
access_key_id => ""
secret_access_key => ""
region => "us-east-1"
bucket => "${BUCKET}"
interval => 60 # seconds
prefix => "${PREFIX}"
type => "cloudtrail"
add_field => { source => gzfiles }
codec => cloudtrail {}
sincedb_path => "/tmp/logstash/cloudtrail"
backup_to_bucket => "${BUCKET}"
backup_add_prefix => "processed-logs/"
delete => true
}
}
filter {
if [type] == "cloudtrail" {
ruby {
'code' => '["responseElements","requestParameters"].each { |field| event.set(field, event.get(field).inspect) }'
}
}
}
output {
if [type] == "cloudtrail" {
elasticsearch {
hosts => ["http://${ES_HOST}:9200"]
index => "cloudtrail-%{+YYYY-MM-dd}"
}
stdout { codec => json }
}
}