Too Many Request 429

I have this cluster configuration:
Server 1: ES node
Server 2: ES node + Logstash

All the instances have pretty much the default settings of the version 2.4
I am retrieving data from social media and then push it to logstash on server 2 via socket. I use server 1 just for query requests. Data from social media are retrieved with a js node application that can push up to 1000 documents/sec into logstash.
In the peak times I get this error in the logstash log:
[logstash.outputs.elasticsearch] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$4@7e188f1b on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@86c6c81[Running, pool size = 4, active threads = 4, queued tasks = 52, completed tasks = 6336583]]"})

What do you suggest? I have many doubts:
How many ES nodes should I have?
It is right to have just one logstash instance? I mean, should I split my data and push it, I don't know, half in a logstash instance and half into another one on a 3rd server? Or should I push all the data into a single logstash instance and then logstash push into several ES nodes?

Are you using bulk requests?

From app to losgstah: a document at a time via socket.
From logstash to elasticsearch: I use the elasticsearch
elasticsearch { hosts => [ "ip:port" ] index => '%{[data][index]}' document_id => '%{[data][post][id]}' document_type => '%{[data][type]}' }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.