I can't quite figure out what is going on here. This error is only popping up for one particular index that I am inserting to. The vast majority of writes from LS to ES are going smoothly. Just to this one particular index it is pooping out.. thought this message was kinda cryptic. Manticore::ClientProtocolException.. not sure what to make of it..
{:timestamp=>"2015-12-02T04:03:03.703000-0800", :message=>"Failed to flush outgoing items", :outgoing_count=>1, :exception=>"Manticore::ClientProtocolException", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:35:ininitialize'", "org/jruby/RubyProc.java:271:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:70:incall'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:245:in call_once'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:148:incode'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/manticore.rb:71:in perform_request'", "org/jruby/RubyProc.java:271:incall'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:190:in perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/manticore.rb:54:inperform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/client.rb:119:in perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/bulk.rb:80:inbulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.7-java/lib/logstash/outputs/elasticsearch/protocol.rb:104:in bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.7-java/lib/logstash/outputs/elasticsearch.rb:542:insubmit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.7-java/lib/logstash/outputs/elasticsearch.rb:541:in submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.7-java/lib/logstash/outputs/elasticsearch.rb:566:inflush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.7-java/lib/logstash/outputs/elasticsearch.rb:565:in flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/buffer.rb:219:inbuffer_flush'", "org/jruby/RubyHash.java:1341:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/buffer.rb:216:inbuffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/buffer.rb:193:in buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/buffer.rb:159:inbuffer_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.7-java/lib/logstash/outputs/elasticsearch.rb:531:in receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/outputs/base.rb:88:inhandle'", "(eval):416:in output_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:244:inoutputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:166:in start_outputs'"], :level=>:warn} {:timestamp=>"2015-12-02T04:22:52.533000-0800", :message=>"Got error to send bulk of actions: ***********************:80 failed to respond", :level=>:error}
this is my logstash config output section:
output{
if "application log" in [tags] {
if "audit log" in [tags]{
elasticsearch {
host => "***"
protocol => "http"
index => "v2-audit-logs"
port => 80
}
}
if [source] == "web_admin" {
elasticsearch {
host => "***"
protocol => "http"
index => "v2-user-events-%{+YYYY.MM.dd}"
port => 80
}
} else if [source] == "phoenix" {
elasticsearch {
host => "***"
protocol => "http"
index => "v2-admin-events-%{+YYYY.MM.dd}"
port => 80
}
}
} else if "twitter" in [tags]{
elasticsearch {
host => "***"
protocol => "http"
index => "twitter-%{+YYYY.MM.dd}"
port => 80
}
} else if "nginx" in [tags] {
elasticsearch {
host => "***"
protocol => "http"
index => "v2-nginx-%{+YYYY.MM.dd}"
port => 80
}
} else {
elasticsearch {
host => "***"
protocol => "http"
port => 80
index => "v2-logstash-%{+YYYY.MM.dd}"
}
}
}
am using port 80 because i'm on AWS ES. almost every elasticsearch output is working fine except for the one pointing to index => "v2-user-events-%{+YYYY.MM.dd}"`