Hello!
I was testing the s3 plugin for a production POC where a Firehose delivery system is delivering Cloudwatch logs into an S3 bucket from where I am reading it with the S3 plugin into logstash
My logstash config is as below:
input {
s3 {
bucket => "****************"
region => "us-east-1"
role_arn => "***********"
type => "s3"
interval => 10
additional_settings => {
"force_path_style" => true
"follow_redirects" => false
}
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
sniffing => false
index => "s3-logs-%{+YYYY-MM-dd}"
}
stdout { codec => rubydebug }
}
As I start up logstash locally, I can see the data reaching to logstash but its not in proper format, like below.
{
"type" => "s3",
"message" => "\u001F�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000͒�n\u00131\u0010�_��\u0015�����x���MC)\u0005D\u0016!**************************************",
"@version" => "1",
"@timestamp" => 2019-07-12T15:32:37.328Z
}
I also tried adding a codec => "gzip_lines" into the configuration, but then logstash was not able to process those files at all. The documentation suggests S3 plugin is supposed to support GZ files out of the box. I was hoping if anyone could point out what I am doing wrong?
Apologies if I didnt post in the right place or with a proper format its my first post here!
Regards,
Arpan