2017-10-20T10:50:59-04:00 INFO Template already exists and will not be overwritten.
2017-10-20T10:50:59-04:00 ERR Failed to perform any bulk index operations: 429 Too Many Requests
2017-10-20T10:50:59-04:00 INFO Error publishing events (retrying): 429 Too Many Requests
"Make sure to watch for TOO_MANY_REQUESTS (429) response codes (EsRejectedExecutionException with the Java client), which is the way that Elasticsearch tells you that it cannot keep up with the current indexing rate. When it happens, you should pause indexing a bit before trying again, ideally with randomized exponential backoff."
how do I pause indexing with randomized exponential backoff?
Also from that elasticsearch page:
"The default index.refresh_interval is 1s, which forces Elasticsearch to create a new segment every second. Increasing this value (to say, 30s) will allow larger segments to flush and decreases future merge pressure"
I shutdown filebeat agents on other nodes and left two running only. Also shutdown elasticsearch nodes that show big count of bulk operation but stil doesn't help. Still see this error in the log:
2017-10-23T10:49:30-04:00 INFO Template already exists and will not be overwritten.
2017-10-23T10:49:30-04:00 ERR Failed to perform any bulk index operations: 429 Too Many Requests
2017-10-23T10:49:30-04:00 INFO Error publishing events (retrying): 429 Too Many Requests
Problem solved..winlogbeat agent was the issue..and I only deployed one winlogbeat agent. I stop the winlogbeat agent and immediately see log indexing from the other two filebeats....but still wondering how one winlogbeat agent brought down the entire elasticsearch cluster (3 master nodes, 25 ingest + data nodes)..Here is what see in the winlogbeat log:
........
017-10-23T10:54:37-04:00 ERR Failed to perform any bulk index operations: Post https://elk-author-prdwi.marketing-int.dm.com:9200/_bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2017-10-23T10:54:37-04:00 INFO Error publishing events (retrying): Post https://elk-author-prdwi.marketing-int.dm.com:9200/_bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
2017-10-23T10:54:38-04:00 INFO Connected to Elasticsearch version 5.4
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.