Logstash failed to push data (create indices) to elasticsearch

Elastic Stack Logstash
1

I am newbie in ELK. I am having a problem to create indices in ES which data coming from logstash (cloudwatch plugin). I use logstash 6.1.4 and ES 6.2.4. The config of logstash likes following

//input {
// cloudwatch {
// access_key_id => "XXXXXX"
// secret_access_key => "YYYY"
// interval => 300
// namespace => "AWS/CloudFront"
// metrics => [ "4xxErrorRate", "5xxErrorRate", "BytesDownloaded", "BytesUploaded", "Requests", "TotalErrorRate" ]
// region => "us-east-1"
// filters => {
// "Region" => "Global"
// }
// add_field => {
// "Region" => "Global"
// "techstack" => "XXXXXXX"
// "source" => "CloudWatch"
// "region" => "us-east-1"
// }
// }
//}
//output {
// elasticsearch {
// hosts => ["127.0.0.1:9200"]
// user => "OOOOOO"
// password => "XXXXXXXX"
// }
//}

I am pretty sure this setup works fine because that config copied from another production ELK farm (the indices are dynamically created). The problem is that I can only find .monitoring-[es|logstash|kibana]-6- and some .watcher-history-7- indcies. However, logstash- is missing. From the logstash log, I see the following

//[2018-05-10T03:34:50,345][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>".monitoring-logstash", //:thread=>"#<Thread:0x6c9b339a@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 sleep>"}
//[2018-05-10T03:34:51,252][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", //:thread=>"#<Thread:0x25b9ae90@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 sleep>"}
//[2018-05-10T03:34:54,505][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>0, "stalling_thread_info"=>{"other"=>[{"thread_id"=>36, "name"=>"[main]<cloudwatch", //"current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in sleep'"}, {"thread_id"=>37, "name"=>"[main]<cloudwatch", //"current_call"=>"[...]/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:89:insleep'"}, ............ {"thread_id"=>35, "name"=>nil, //"current_call"=>"[...]/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:133:in `initialize'"}]}}

How could I fix this problem?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.