We are having some problems with discovery when restarting nodes. At times when nodes were restarted most would join into one large cluster, but then a few would form their own mini-clusters of one or two nodes. Whenever this happens we get eight or so of Zen ping warnings. If we get two or fewer of these warnings the node usually joins the cluster correctly.
[2011-02-25 09:06:35,377][WARN ][discovery.zen.ping.multicast] [Piaget, Jean] received ping response with no matching id 
The mini-clusters get "State not recoverable" and "disable_persistence_state: true" under blocks.
If somebody can point us to some common Zen discovery "gotchas" it would be much appreciated.