Too_many_requests (429)

[hungl99]

I observed these errors from the filebeat log.

2017-10-20T10:50:59-04:00 INFO Template already exists and will not be overwritten.
2017-10-20T10:50:59-04:00 ERR Failed to perform any bulk index operations: 429 Too Many Requests
2017-10-20T10:50:59-04:00 INFO Error publishing events (retrying): 429 Too Many Requests

Here is what I found from https://www.elastic.co/guide/en/elasticsearch/reference/5.6/tune-for-indexing-speed.html

"Make sure to watch for TOO_MANY_REQUESTS (429) response codes (EsRejectedExecutionException with the Java client), which is the way that Elasticsearch tells you that it cannot keep up with the current indexing rate. When it happens, you should pause indexing a bit before trying again, ideally with randomized exponential backoff."

how do I pause indexing with randomized exponential backoff?

Also from that elasticsearch page:
"The default index.refresh_interval is 1s, which forces Elasticsearch to create a new segment every second. Increasing this value (to say, 30s) will allow larger segments to flush and decreases future merge pressure"

Where can I set the " index.refresh_interval"

[hungl99]

@hungl99

Failed to perform any bulk index operations: 429 Too Many Requests

Here is a snapshot of thread_pool which show heavy bulk index operation. I'm not sure how to handle these without affecting indexing.

[hungl99]

I shutdown filebeat agents on other nodes and left two running only. Also shutdown elasticsearch nodes that show big count of bulk operation but stil doesn't help. Still see this error in the log:

2017-10-23T10:49:30-04:00 INFO Template already exists and will not be overwritten.
2017-10-23T10:49:30-04:00 ERR Failed to perform any bulk index operations: 429 Too Many Requests
2017-10-23T10:49:30-04:00 INFO Error publishing events (retrying): 429 Too Many Requests

[hungl99]

Problem solved..winlogbeat agent was the issue..and I only deployed one winlogbeat agent. I stop the winlogbeat agent and immediately see log indexing from the other two filebeats....but still wondering how one winlogbeat agent brought down the entire elasticsearch cluster (3 master nodes, 25 ingest + data nodes)..Here is what see in the winlogbeat log:
........
017-10-23T10:54:37-04:00 ERR Failed to perform any bulk index operations: Post https://elk-author-prdwi.marketing-int.dm.com:9200/_bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2017-10-23T10:54:37-04:00 INFO Error publishing events (retrying): Post https://elk-author-prdwi.marketing-int.dm.com:9200/_bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2017-10-23T10:54:38-04:00 INFO Connected to Elasticsearch version 5.4