Filebeats behavior when new ES node elected up to Primary

[rachelkelly]

Hello, we are very close to the first version of our Elasticsearch, rather a proof of concept. For this basic version I'm trying to do:
filebeats (application logs, http logs, /var/log*) on a handful of servers (total is only about 30 - this is just POC)
to
several elasticsearch nodes (a handful of nodes on an ECS cluster with 2 EC2 instances on AWS)
to
kibana

My question, how do filebeats behave when a new Elasticsearch Node is elected Primary? Can someone point me to the technical spec on how exactly it ships to a different, newly Primary node in the event of node failure?

FYI: 1 - Due to regulatory concerns we are not able to use AWS' Elasticsearch Service. 2 - We plan on incorporating Logstash later. 3 - It is an open question whether or not we will incorporate a message queuer like Kafka or Redis to manage the shipping between filebeat agents and Elasticsearch

[Christian_Dahlqvist]

Filebeat can send data to any node in the Elasticsearch cluster that it has configured. Data does not have to go through the master node. While a master node is being elected, the cluster will however not accept writes, but once a master is in place Filebeat will retry and continue sending data to all nodes in the cluster.

[rachelkelly]

Hi Christian, thanks for your answer. To be super clear, you're saying that beats ship to EVERY ES node (unless it is actively being elected)? Do you have links on bandwidth usage of filebeats? Seems like with every new node instantiated for ES (for scaling etc), network traffic will increase by that amount - is that accurate?

[Christian_Dahlqvist]

Filebeat usually has multiple nodes configured, typically all data nodes, and will distribute the request load across these nodes. Each event is however sent to only one of the ES node (not all), unless there is a failure and Filebeat need to retry on a different node. If a master fails and a new one need to be elected, the cluster may temporarily stop accepting written for a short period of time until the election has completed. During this time Filebeat will hold the events and retry until it succeeds. Is that clearer?