Hi All,
Hoping someone might be able to help me debug my logstash/S3/cloudtrail issue. This is my first time trying to set this up but I am not making any progress.
My setup :
RHEL 7 logstash 2.3 running in AWS on EC2. My cloudtrail logs are in S3
cat /etc/logstash/conf.d/cloudtrail.conf (keys have been removed)
input {
s3 {
bucket => "xyz_cloudtrail"
access_key_id => '.......'
secret_access_key => '........'
delete => false
codec => cloudtrail {}
interval => 10 # seconds
region => "ap-southeast-2"
}
}
output {
stdout { codec => cloudtrail }
elasticsearch {
hosts => "localhost:9200"
index => "cloudtrail"
sniffing => true
manage_template => false
}
}
no idea if this is correct or not, can anyone advise ?
When I run ...
/opt/logstash-2.3.2/bin/logstash --config /etc/logstash/conf.d/ -l /var/log/logstash/logstash.log --debug
i can see this message..
{:timestamp=>"2016-05-24T09:51:02.716000+1000", :message=>"Connection refused", :class=>"Manticore::SocketException", :backtrace=>["/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/manticore-0.5.5-java/lib/manticore/response.rb:37:in initialize'", "org/jruby/RubyProc.java:281:incall'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/manticore-0.5.5-java/lib/manticore/response.rb:79:in call'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/manticore-0.5.5-java/lib/manticore/response.rb:256:incall_once'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/manticore-0.5.5-java/lib/manticore/response.rb:153:in code'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/transport/http/manticore.rb:84:inperform_request'", "org/jruby/RubyProc.java:281:in call'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/transport/base.rb:257:inperform_request'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/transport/http/manticore.rb:67:in perform_request'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/transport/sniffer.rb:32:inhosts'", "org/jruby/ext/timeout/Timeout.java:147:in timeout'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/transport/sniffer.rb:31:inhosts'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/transport/base.rb:79:in reload_connections!'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.6.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:72:insniff!'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.6.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in start_sniffing!'", "org/jruby/ext/thread/Mutex.java:149:insynchronize'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.6.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in start_sniffing!'", "org/jruby/RubyKernel.java:1479:inloop'", "/opt/logstash-2.3.2/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.6.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:59:in `start_sniffing!'"], :level=>:error}
Can anyone provide any assistance on where I am going wrong or what needs to be changed or how to troubleshoot this further ?
btw: I can download S3 logs manually using s3cmd so would think it's not a permission issue