Elasticsearch Output - Request size exceeded error

Elastic Stack Logstash
1

Hello everyone,

Occasionally we've been seeing our logstash server stop processing logs from our redis queue and start to show some errors in its logs about request size being too large. This causes our logstash server to stop working until a restart is done on the logstash_server service. After that it will start to process logs again.

We're using the AWS Elasticsearch service as our ES backend and they have a limit of 10485760 bytes per request. I've tried lowering the flush_size option in our elasticsearch config to 50 to see if that would help but we still see this occurring periodically, about once a week right now. I'm going to try to lower it 20 but wanted to see if there's anywhere I can check to get more information on the batch of logs being sent so I can see where the offending log is coming from. Here's some more information on our logstash server and elasticsearch setup:

Logstash Version: logstash 2.3.4

Elasticsearch Output Plugin Config:

output {
  elasticsearch {
    'hosts' => ["logstash-elasticsearch.example.com:80"]
    'flush_size' => 50
  }
}

Logs of the Error:

{:timestamp=>"2016-07-25T16:50:02.153000+0000", :message=>"[413] {\"Message\":\"Request size exceeded 10485760 bytes\"}", :class=>"Elasticsearch::Transport::Transport::Errors::RequestEntityTooLarge", :backtrace=>["/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:201:in `__raise_transport_error'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:312:in `perform_request'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:128:in `perform_request'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/bulk.rb:90:in `bulk'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in `each_slice'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:129:in `worker_multi_receive'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:114:in `multi_receive'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:232:in `worker_loop'", "/opt/logstash/server/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:warn}
{:timestamp=>"2016-07-25T16:50:06.230000+0000", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://logstash-elasticsearch.example.com\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Broken pipe", :class=>"Manticore::SocketException", :level=>:error}

Thanks in advance,

Steve

2

Does you elasticsearch run over port 80? Can you paste the entire config of logstash.

3

yes, we run our elasticsearch service over port 80. Here's the entire logstash config that we are using at the moment:

input {
  redis {
    'data_type' => "list"
    'host' => "logstash-redis.example.com"
    'key' => "logstash"
    'type' => "redis_input"
  }
}

input {
  file {
    type => "logstash-log"
    path => "/opt/logstash/server/log/logstash.log"
    start_position => "beginning"
  }
}

input {
  http {
    port => 19300
    user => user
    password => "XXXXXXXXXXXXX"
  }
}

filter {
  if [type] == "logstash-log" {
    ruby {
      code => "event['message'] = event['message'].gsub('=>', '=').gsub(':message', ':log_message')"
    }

    kv {
      add_field => {"app" => "logstash-server"}
      trimkey => "{:"
      trim => "}\""
      field_split => ", "
    }
  }
}

filter {
  if [type] == "rails" {
    ruby {
      'code' => "event['rails_params'] = event['params'].to_s if event.include?('params') && !event['params'].empty?"
      'remove_field' => ["params"]
    }
  }
}

output {
  elasticsearch {
    'hosts' => ["logstash-elasticsearch.example.com:80"]
    'flush_size' => 50
  }
}
4

Can you add a protocol => http to it and give it a try.

5

It doesn't look like there is a protocol setting in the elasticsearch output plugin in logstash version 2.3.4. From their documentation:

This output only speaks the HTTP protocol. HTTP is the preferred protocol for interacting with Elasticsearch as of Logstash 2.0.

Is there something I'm missing there? Thanks for the response again.

1 Like
6

My bad. In the previous versions of LS there was a setting to manually specify the protocol. But seems like it is default now.

7

Maybe you could do something in a ruby filter where you check the total size and add a metadata field if it is over a certain size, and conditionally write to a file output if that metadata field is set, else proceed with your normal output.

Just spitballin' here.

8

+1, same problem

9

+1, same problem as well.