Issue logstashing to Found - Continued discussion

[cstrzadala (Christian Strzadala)]

Continuing the discussion from Issue logstashing to Found:

@James_Tang

Got error logstashing (Logstash 1.5.4) to Elastic Search on Found.

The following was thrown out in the debug logs and I am at a lost as to what is happening. I tried to adjust the flush sizes, idle_flush time and also the java heap size. But it did not help

It was working until a reboot, but the connectivity to Elasticsearch/Found is still valid and a curl test informed that there is no connectivity issue with the network. Any suggestions?

Failed to flush outgoing items {:outgoing_count=>20, :exception=>"Manticore::ClientProtocolException", :backtrace=>["/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:35:in `initialize'", "org/jruby/RubyProc.java:271:in `call'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:70:in `call'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:245:in `call_once'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:148:in `code'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/manticore.rb:71:in `perform_request'", "org/jruby/RubyProc.java:271:in `call'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/base.rb:190:in `perform_request'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/transport/http/manticore.rb:54:in `perform_request'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.12/lib/elasticsearch/transport/client.rb:119:in `perform_request'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.12/lib/elasticsearch/api/actions/bulk.rb:80:in `bulk'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.7-java/lib/logstash/outputs/elasticsearch/protocol.rb:104:in `bulk'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.7-java/lib/logstash/outputs/elasticsearch.rb:542:in `submit'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.7-java/lib/logstash/outputs/elasticsearch.rb:566:in `flush'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1341:in `each'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/buffer.rb:216:in `buffer_flush'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/buffer.rb:159:in `buffer_receive'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-1.0.7-java/lib/logstash/outputs/elasticsearch.rb:531:in `receive'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/outputs/base.rb:88:in `handle'", "(eval):73:in `output_func'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:244:in `outputworker'", "/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:166:in `start_outputs'"], :level=>:warn, :file=>"stud/buffer.rb", :line=>"231", :method=>"buffer_flush"}

elasticsearch {
cluster => "45cd54"
host => "xxx.us-east-1.aws.found.io"
port => "9243"
ssl => "true"
ssl_certificate_verification => "true"
protocol => "http"
user => "user"
flush_size => "20"
idle_flush_time => "5"
password => "password"
}

[James_Tang]

Applying an approach from Sherlock Holmes, rule out the possibilities. So if anyone can help with the below questions?

1. Can logstash and ES on Found be able to handle ipv6 transport? ie receiving logs logstashing into Found?
   https://forums.aws.amazon.com/thread.jspa?messageID=536049. No definitive answers based on the fourm. Last update was recent enough.
2. How can we rule out that it is an network issue locally? Will using curl be sufficient? Any recommendations as to what kind of test can we performed to rule them out? Especially when the host has both IPV4 and 6 addresses.

Additional information:

• Switching the configuration to ssl=> false is not working. The same exception is happening.

Failed to flush outgoing items {:outgoing_count=>1, :exception=>"Manticore::ClientProtocolException", :backtrace=>["/volume1/apps/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:35:in `initialize'",
.
.
.
.

regards