I manage my clusters (v2.4) on AWS using CloudFormation. I have dedicated data nodes, a dedicated master, and a load balancer in front of the data nodes for the convenience of having one URL to reach the cluster.
Recently I updated from Debian 8.4 AMIs to 8.6 AMIs, and suddenly the clusters stopped coming up properly. There are no errors in the logs, and the nodes seem to be discovering each other and the master node, but the load balancer can no longer register the data nodes as being healthy (using http://node-internal-dns:9200/_cluster/health), and trying to hit the data nodes with curl on port 9200 yields an immediate "Connection refused". My security groups, networking, etc. didn't change ... only the switch to Debian 8.6 AMI.
My data node configuration looks like this:
cluster.name: XXX
node.master: false
plugin.mandatory: cloud-aws
http.host: _ec2:privateDns_
network.host: _ec2:privateIpv4_
discovery.type: ec2
discovery.zen.ping.multicast.enabled: false
discovery.zen.minimum_master_nodes: 1
cloud.aws.region: us-west-2
cloud.aws.groups: sg-XXX
How can I go about debugging this problem?