ClientProtocolException

Elastic Stack Logstash
1

Hi all

I'm writing to about 5 sets of indices from logstash. One of those writes is failing for some reason. Event data coming in fairly sporadically (a couple hundred a day).
Seems like for every event that tries to come into the pipeline we see this --
{:timestamp=>"2015-12-02T04:03:03.703000-0800", :message=>"Failed to flush outgoing items", :outgoing_count=>1, :exception=>"Manticore::ClientProtocolException", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.4.4-java/lib/manticore/response.rb:35:ininitialize'",...
followed by this:
{:timestamp=>"2015-12-02T04:22:52.533000-0800", :message=>"Got error to send bulk of actions: ***********************:80 failed to respond", :level=>:error}

Is very strange, the majority of my writes are working seamlessly. Can't quite figure out what is special about this particular set of indices. Port 80 is fine here, I'm working with AWS ES service which is hosted on 80.

2

What version are you on?

3

[root@elk-elk-esclient-01 ~]$ /opt/logstash/bin/logstash --version logstash 1.5.4
[root@elk-elk-esclient-01 ~]$ alternatives --config java
There are 2 programs which provide 'java'.
Selection Command
-----------------------------------------------
`* 1 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85-2.6.1.2.el7_1.x86_64/jre/bin/java

  • 2 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.65-2.b17.el7_1.x86_64/jre/bin/javaEnter to keep the current selection[+], or type selection number:`

ES version i'm pretty sure is 1.5.x.. it is AWS ES service and apparently you can't checkout that actual version of ES they are running for you.

Again, just to reiterate bulk of my writes are fine. This index pattern I'm writing to has relatively low traffic but not as low as some of the other index patterns.. also when I refer to index pattern I pretty much just mean foo-%{+YYYY.MM.dd}.

4

Well without access to your ES logs it's probably going to be hard to say.

5

OK. submitted an AWS support ticket, i'll keep this updated with findings from there.