Logstash HTTP output plugin logging vague "Manticore::ClientProtocolException" errors

Elastic Stack Logstash
1

For all general issues, please provide the following details for fast resolution:

  • Version: logstash 2.4.0
  • Operating System: Ubuntu 14.04.5 LTS
  • Config File:
output {
  http {
    http_method => "post"
    url => "http://logs-01.loggly.com/inputs/{{ loggly_api_token }}/tag/http/"
    format => "message"
    message => "%{message}"
    headers => [
        "X-LOGGLY-TAG", "%{appId},%{type},instance_{{ instance_id }},environment_{{ instance_environment }},purpose_{{ instance_purpose }},{{ instance_hostname }}"
    ]
  }
}

( The {{ ... }} are filled in by jinja)

I'm using the http_client plugin to send to loggy (the loggly plugin isn't as nice :frowning: ). Once in a while (like every couple minutes) we see an error like this.

{:timestamp=>"2016-09-28T17:30:01.180000+0000", :message=>"logs-01.loggly.com:80 failed to respond", :url=>"http://logs-01.loggly.com/inputs/3429137e-2e3e-4729-a767-dc3040a22c9f/tag/http/", :method=>:post, :body=>"2016-09-28T17:30:00.800+0000 [conn57] getmore urban_compass_development.boston_images cursorid:893414142805 ntoreturn:0 keyUpdates:0 numYields:299 locks(micros) r:676326 nreturned:607 reslen:4197280 587ms", :headers=>{"X-LOGGLY-TAG"=>"mongodb", "Content-Type"=>"text/plain"}, :class=>"Manticore::ClientProtocolException", :backtrace=>nil, :level=>:error}

I'm unable to reproduce this issue with curl, and a quick google search turned up https://github.com/logstash-plugins/logstash-output-http/issues/20

Any suggestions? I'm going to try first enabling retry_non_idempotent but I wasn't sure what exactly behavior that has and if there's a chance of duplicate logs.