I'm trying to use LS2.1.1 with ES2.1.1 for AWS CloudTrail Log Analysis from S3 Bucket using logstash-input-s3 and logstash-codec-cloudtrail plugin.
I am facing problem when trying to start my Logstash service where as my configuration test passed successfully.
For debugging the problem I tried the --debug command and got the below error.
Conmmand : /opt/logstash/bin/logstash -f logstash-s3.conf –debug
Part of error
Settings: Default filter workers: 1
Registering s3 input {:bucket=>"cloudtrailbucket", :region=>"ap-southeast-1", :level=>:info, :file=>"logstash/inputs/s3.rb", :line=>"78", :method=>"register"}
The error reported is:
can't convert Symbol into Integer
org/jruby/RubyString.java:3919:in []=' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-mixin-aws-2.0.2/lib/logstash/plugin_mixins/aws_config/v1.rb:40:inaws_options_hash'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.3/lib/logstash/inputs/s3.rb:397:in get_s3object' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-s3-2.0.3/lib/logstash/inputs/s3.rb:80:inregister'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:165:in start_inputs' org/jruby/RubyArray.java:1613:ineach'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:164:in start_inputs' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:100:inrun'
/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/agent.rb:165:in execute' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/runner.rb:90:inrun'
org/jruby/RubyProc.java:281:in call' /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/runner.rb:95:inrun'
org/jruby/RubyProc.java:281:in call' /opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24:ininitialize'
My configuratoion used for logstash-input-s3:
input {
s3 {
bucket => "cloudtrailbucket"
delete => false
interval => 60 # seconds
prefix => "AWSLogs//CloudTrail/ "
type => "cloudtrail"
codec => "cloudtrail"
region => "ap-southeast-1"
aws_credentials_file => "/etc/logstash/conf.d/s3_credentials.ini"
sincedb_path => "/opt/logstash_cloudtrail/sincedb"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "Client-Cloudtrail"
}
}
Has anyone done this before and can share the steps.