Life was beautiful on a single node but I index 300gb of logs per day and after ten or eleven days I added a new storage node. The new node doesnt work. I had high hopes for the cluster to automagically provision it but I guess it needs help or I need to change my expectations.
I am using ES 2.1.1 on java 1.7
The nodes are on the same same ip subnet. The original node seems to see the new node but the new node complains about missing indexes and has other errors including messages about a node leaving. I can ping and connect to port 9200 & 9300 from each server to the other. There are no firewalls in between.
I have blindly included some logs and data from the cluster in hopes they make sense to someone with a clue. Does anyone know what is going wrong here?
Cluster status information: http://pastebin.com/VWz9jNU7
Logs from new node: http://pastebin.com/wvS8SBZ1