ERROR: Too Many Requests - rejected execution of org.elasticsearch.transport.TransportService on EsThreadPoolExecutor

[agonzalez (Alberto)]

during peak time once a day i may have +3000 clients indexing documents using NEST client and i see lot of connection fails with these errors. are both related? how can increase resources to avoid this? just have one node ES 5.2.

# ServerError: ServerError: 429Type: es_rejected_execution_exception Reason: "rejected execution of org.elasticsearch.transport.TransportService$7@16399044 on EsThreadPoolExecutor[index, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@49aa71ba[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 1521828]]"
# OriginalException: System.Net.WebException: The remote server returned an error: (429) Too Many Requests.

and also see this

# OriginalException: System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x40645d20 + 0x00197> in <filename unknown>:0 
  at System.Net.HttpWebRequest.GetResponse () <0x4063c200 + 0x00053> in <filename unknown>:0 
  at Elasticsearch.Net.HttpConnection.Request[TReturn] (Elasticsearch.Net.RequestData requestData) <0x4062a000 + 0x0029f> in <filename unknown>:0 
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>

[warkolm (Mark Walkom)]

You will need to add more resources to your cluster.
More nodes or increase the size of the one you have.

[agonzalez (Alberto)]

But the node has 4CPUs and 8GB RAM. maximum cpu usage is 60% during that peak time the rest of the day sever is idle and memory usage now is:

# free -m
              total        used        free      shared  buff/cache   available
Mem:           7823        2936         481           3        4404        4605
Swap:          2047         161        1886

what should i increase? just memory? do i need to change any config settings to increase limits?