yeah, I think we have figured something out on this..
All of my nodes are set at MTU 9000 in centos but the ports on the switch are at MTU 2500 for the physical nodes..
I am adjusting the data node elk02 to MTU 1500 right now to see if that helps. ELK01 is a VM that has full support for jumbo frames all the way through, so keeping at 9000 right now.
Will update as soon as I know more.