Filebeat reports intermittent errors, sometimes successful, sometimes not!

Here is the output of docker logs!
The original version 6.4.1 had this error, but now I change to 6.6.1 and still have this problem.
Please help me analyze it, thank you!

2019-02-27T08:01:31.471+0700	ERROR	pipeline/output.go:121	Failed to publish events: temporary bulk send failure
2019-02-27T08:01:31.471+0700	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(https://search-bidogo-test1-suhxwkzrfh66xyyipktsvbanei.ap-southeast-1.es.amazonaws.com:443))
2019-02-27T08:01:31.471+0700	INFO	[publish]	pipeline/retry.go:189	retryer: send unwait-signal to consumer
2019-02-27T08:01:31.471+0700	INFO	[publish]	pipeline/retry.go:191	  done
2019-02-27T08:01:31.471+0700	INFO	[publish]	pipeline/retry.go:166	retryer: send wait signal to consumer
2019-02-27T08:01:31.471+0700	INFO	[publish]	pipeline/retry.go:168	  done
2019-02-27T08:01:31.474+0700	INFO	elasticsearch/client.go:721	Connected to Elasticsearch version 6.3.1
2019-02-27T08:01:31.485+0700	INFO	template/load.go:130	Template already exists and will not be overwritten.
2019-02-27T08:01:31.485+0700	INFO	instance/beat.go:894	Template successfully loaded.
2019-02-27T08:01:31.485+0700	INFO	pipeline/output.go:105	Connection to backoff(elasticsearch(https://search-bidogo-test1-suhxwkzrfh66xyyipktsvbanei.ap-southeast-1.es.amazonaws.com:443)) established
2019-02-27T08:01:31.485+0700	INFO	[publish]	pipeline/retry.go:189	retryer: send unwait-signal to consumer
2019-02-27T08:01:31.485+0700	INFO	[publish]	pipeline/retry.go:191	  done
2019-02-27T08:01:33.311+0700	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":35080,"time":{"ms":15}},"total":{"ticks":151820,"time":{"ms":25},"value":151820},"user":{"ticks":116740,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral_id":"38830e86-746e-4f00-b82d-975c6d0a4a2b","uptime":{"ms":53100904}},"memstats":{"gc_next":22571280,"memory_alloc":15051440,"memory_total":7626813672}},"filebeat":{"harvester":{"open_files":10,"running":10}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":236,"batches":9,"failed":26,"total":262},"read":{"bytes":70205},"write":{"bytes":160203}},"pipeline":{"clients":15,"events":{"active":4121,"retry":233}}},"registrar":{"states":{"current":10}},"system":{"load":{"1":1.85,"15":2.58,"5":2.67,"norm":{"1":0.925,"15":1.29,"5":1.335}}}}}}
2019-02-27T08:01:38.084+0700	ERROR	pipeline/output.go:121	Failed to publish events: temporary bulk send failure
2019-02-27T08:01:38.084+0700	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(https://search-bidogo-test1-suhxwkzrfh66xyyipktsvbanei.ap-southeast-1.es.amazonaws.com:443))
2019-02-27T08:01:38.085+0700	INFO	[publish]	pipeline/retry.go:166	retryer: send wait signal to consumer
2019-02-27T08:01:38.085+0700	INFO	[publish]	pipeline/retry.go:168	  done
2019-02-27T08:01:38.087+0700	INFO	elasticsearch/client.go:721	Connected to Elasticsearch version 6.3.1
2019-02-27T08:01:38.093+0700	INFO	template/load.go:130	Template already exists and will not be overwritten.
2019-02-27T08:01:38.093+0700	INFO	instance/beat.go:894	Template successfully loaded.
2019-02-27T08:01:38.093+0700	INFO	pipeline/output.go:105	Connection to backoff(elasticsearch(https://search-bidogo-test1-suhxwkzrfh66xyyipktsvbanei.ap-southeast-1.es.amazonaws.com:443)) established
2019-02-27T08:01:38.094+0700	INFO	[publish]	pipeline/retry.go:189	retryer: send unwait-signal to consumer
2019-02-27T08:01:38.094+0700	INFO	[publish]	pipeline/retry.go:191	  done
2019-02-27T08:01:49.223+0700	ERROR	pipeline/output.go:121	Failed to publish events: temporary bulk send failure
2019-02-27T08:01:49.223+0700	INFO	pipeline/output.go:95	Connecting to backoff(elasticsearch(https://search-bidogo-test1-suhxwkzrfh66xyyipktsvbanei.ap-southeast-1.es.amazonaws.com:443))
2019-02-27T08:01:49.223+0700	INFO	[publish]	pipeline/retry.go:166	retryer: send wait signal to consumer
2019-02-27T08:01:49.223+0700	INFO	[publish]	pipeline/retry.go:168	  done
2019-02-27T08:01:49.225+0700	INFO	elasticsearch/client.go:721	Connected to Elasticsearch version 6.3.1
2019-02-27T08:01:49.233+0700	INFO	template/load.go:130	Template already exists and will not be overwritten.
2019-02-27T08:01:49.233+0700	INFO	instance/beat.go:894	Template successfully loaded.
2019-02-27T08:01:49.233+0700	INFO	pipeline/output.go:105	Connection to backoff(elasticsearch(https://search-bidogo-test1-suhxwkzrfh66xyyipktsvbanei.ap-southeast-1.es.amazonaws.com:443)) established
2019-02-27T08:01:49.234+0700	INFO	[publish]	pipeline/retry.go:189	retryer: send unwait-signal to consumer
2019-02-27T08:01:49.234+0700	INFO	[publish]	pipeline/retry.go:191	  done
2019-02-27T08:01:51.258+0700	INFO	[publish]	pipeline/retry.go:166	retryer: send wait signal to consumer
2019-02-27T08:01:51.259+0700	INFO	[publish]	pipeline/retry.go:168	  done

Could you please share your debug logs and configuration formatted using </>?
It is possible that sometimes ES gets a too big bulk request from your Filebeat instance. Thus, the request gets rejected. I think you can get more info from this article: https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster

Thank you. That could be the problem!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.