I have built a test environment with 1 master node, 2 data/master nodes, and 2 logstash client nodes (no master / no data).
The logstash client nodes are configured to send output using elasticsearch with the http protocol and send to the local host. I have elasticsearch on each of the logstash servers running and pointing to the 3 servers (master, data1, and data2).
host => "logstash1"
cluster => "elasticsearch"
protocol => "http"
When I look at the elasticsearch log on the 2 logstash servers, I see 'disconnected from [[master] due to explicit disconnect call. When I look at the master, I see '[[master] timed out waiting for all nodes to process published state ...
The logstash nodes keep getting dropped and then added again about once or twice every two hours. Is this normal behavior?
There is a firewall between the logstash nodes and the ES servers. I have read that can cause issues, but I have tried setting the timeout window thresholds higher and setting up a heartbeat plugin. Nothing has stopped the disconnects.
What am I missing? Is there a better way to do this?