Logstash cloudtrail codec doesn't work

[niraj_kumar]

Hi,

I am trying to user the logstash codec cloudtrail to ingest cloudtrail data but that doesn't seem to work.

16:27:00.265 [[main]<file] WARN  logstash.codecs.cloudtrail - Received an event that has a different character encoding than you configured. {:text=>"\\u001F\\x8B\\b\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\xED\\x99[\\x93\\xA3\\xB6\\xB6\\x80\\xFF\\xCAT\\xBF:\\xB4\\x91\\u0010\\xB7~\\u0003\\u00036\\xD8\\xC6\\u0018\\x8Cm\\xC8ٵK\\x80\\xF0\\x8D\\x8B\\xB9\\xD9ة\\xF9\\xEF[\\xEE\\x9Ed&ɞ\\x9C\\xEC\\xDAg\\xEA\\xA4:\\xD3\\u000F]\\u0005ZZҺ\\xE8[\\v\\xEB\\xA7'\\x97\\xC4e\\x9D4O/?\\xFE\\xF4D.\\xA4hפn\\u000Ee\\xF1\\xF4\\xF2\\u0004\\x9EY\\xFE釧\\xAE!\\xB5\\x99БC{{z\\xF9驽\\x9D\\t\\u001DU6\\x9E\\u0012\\xC7eW\\xB4T\\xE6\\\\\\u001F\\x8A\\xF8pƙ\\x99<\\x86LM\\xB1|\\xC4/\\xF9\\x8D\\xBD\\u0010\\u0015g:R\\xA6:\\x95\\xC2o\\xF2\\xAF2\\b\\u0002I\\xE2y\\x99\\xE5Y\\xF6\\xE9\\xE3\\u000Fok\\xAF\\u000E\\xF9C5d\\x81Ȱ\\\"\\u0003\\xD9\\u0015@/\\x90}a\\xF9\\xF0铈Wvu\\xFC\\u0010j\\xDA\\xE6\\u0019\\xE7\\xF8^\\u0016\\xF8\\xDA<\\xC7e\\xFE\\xB3\\x84\\x8D_\\x95(M\\xD3\\xE5\\xC4-3\\xF2X\\xF9ڸd\\xF7fV\\xD70\\u00047-\\u0003\\xE8\\xFB\\xE6U\\x9B\\xE9(IR\\x93\\xA6y\\u0018\\xCD?\\v\\xFC3\\xE4\\xE13@\\x9F\\x8CWvT+\\u001DR˶\\u001C\\xC2g$<\\x83\\u000Fέݗ\\u0005}\\u0012\\x9F\\x85\\u000F\\xB3C\\xD1\\xF5C\\xEE\\u0019p\\xCF,\\u0003\\x80\\xCC\\xD0\\t\\xA4>\\xC4t~M\\xAA\\x8E4\\xAD\\x83k\\xBA\\xAB\\x96\\xBA\\xF6\\xE1\\xC1\\x9AnJ\\xA9\\u001F{\\xC1u\\xF1B\\xF7\\xF6r\\xC0\\xF9\\v5\\x94\\u0015\\u0001\\v9\\u0019\\xC9\\xE8\\xE5!3t\\tN\\u0016Ev{(\\xA2\\xCF\\u001E\\xDD\\\"\\xB5ᓁ!\\xE9\\x9A\\u007F\\x9AE\\xD3\\xE2\\\"&\\xFF\\u001C\\xD5\\xF8\\x9A\\x91\\xFA\\xE1Jjɹ,\\u001A\\xA2g$\\xA7;\\u007F]2\\xAE\\xC9k\\bq\\xF6\\xFAH#AuM\\xC9\\xED-^\\x9E\\xA9X\\xB3\\xD9L\\x98\\xAB3o\\xA1o|m\\xBC|x\\xB3?\\u001Fjܾy\\xCD\\xEA\\xB2\\u000F\\x90\\xFD\\xE1\\xC3#6\\u001F\\xB8\\xB7\\xA0|p\\xE6\\u000F\\u001F\\xBE\\xEDjU\\x9E\\xC8C\\xD0X\\x96Z\\xB0M\\xEE\\xBA9\\xFC\\xFCwձ\\xB6\\xC8\\xC7\\xEA\\u000E\\x9EOH\\x8B\\xBBEq\\xEBo\\xB3@ٲ\\xD3S\\xD9\\v\\xF6QA\\xA1\\x9AՓ\\u001A\\xE3jd\\u000E\\xE2L\\xBB\\xB4\\u0005&\\xEBC\\xB5:lG*[\\f\\x8D\\xF1\\xBA\\u0011\\xC5F>zl\\xD9t\\u001E\\x92\\u05ED|\\xC9\\xE3\\xFD\\xCE=n\\x8D \\xD8\\f\\x8E2nʪ\\u001Fρє*\\u00199Zf\\u0004nid\\x97\\xBD;\\xEE{\\xC9=", :expected_charset=>"UTF-8"}
16:27:00.268 [[main]<file] DEBUG logstash.inputs.file - observe_read_file: general error reading /data/bigdata/AWSLogs/444444444444/CloudTrail/us-east-1/2017/07/20/444444444444_CloudTrail_us-east-1_44444444444444Z_FVbdjILbNstsDCwS.json.gz - error: #<LogStash::Json::ParserError: Unexpected character ('\' (code 92)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

My Config as of now:

input {
file {
        path => "/data/**/*.gz"
        codec => "cloudtrail"
        start_position => "beginning"
#       sincedb_path => "/usr/share/logstash/.sincedb"
        sincedb_path => "/var/lib/logstash/.sincedb"
        type => "cloudtrail"
        max_open_files => "1024"
  }
}

filter {
   grok {
       match => {"path" => "/data/(?<tstmp>\S+)/.*"}
   }
}

output {
  stdout { codec => json }

  elasticsearch {
        hosts => ["xx.xx.xx.xx:9200"]
        index => "%{[tstmp]}-%{+YYYY-MM}"
   }
}

Has anyone ever made cloudtrail work with logstash. I seem to have been stuck with this thing

--
Niraj

[nfplatzke]

Here is my config that works

 input {
     s3 {
         region => "us-east-1"
         bucket => "logbucket"
         prefix => "cloudtrail/AWSLogs/account_number/CloudTrail/"
         type => "cloudtrail"
         add_field => { source => gzfiles }
         codec => cloudtrail {}
         sincedb_path => "/tmp/logstash/cloudtrail"
         backup_to_bucket => "logbucket"
         backup_add_prefix => "processed-logs/"
         delete => true
     }
 }
 filter {
     if [type] == "cloudtrail" {
         ruby {
             'code' => '["responseElements","requestParameters"].each { |field| event.set(field, event.get(field).inspect) }'
        }
     }
 }
 output {
     if [type] == "cloudtrail" {
         elasticsearch {
             hosts => "http://logstash:9200"
             index => "logstash-cloudtrail-%{+YYYY-MM-dd}"
         }
     }
 }

[niraj_kumar]

@nfplatzke Thanks a lot. I will give it a try.