Multiple nodes on elasticsearch

The general form of this message suggests that the master node is failing to connect back to the data node at its advertised address and is timing out after 30 seconds in its attempt to do so, which still suggests a networking misconfiguration.

It's hard to give any more precise feedback since this single line from the log has been post-processed since it was written by Elasticsearch. It's impossible to know which of the strange things in this line are the effects of the post-processing and which are related to your actual problem. For instance, the node names do not match up:

datanode1-ac020034.com vs datanode1-ac020034.phx.aexp.com
node-ac020035.com vs node-ac020035.phx.aexp.com

Also some of the IP addresses are 10.x.x.x and some are 172.x.x.x which is unusual and might or might not be the source of the problem.

Also this line would have been followed by a stack trace that would help to explain the situation in more detail.