We're using an external marvel cluster to monitor our production elasticsearch cluster. As part of rolling out elasticsearch 2.3.1 I deployed it to the external marvel cluster but have not yet upgraded the production cluster. That has left us in a state where the production cluster is on 2.2.0 and the marvel cluster is on 2.3.1 with kibana 4.5.0.
At this point I'm only seeing the "Waiting for marvel data" screen when trying to use Marvel. I can confirm that there are marvel documents being indexed in the .marvel-es-2016.04.06 index as I can see them via kibana. There are also no errors in the elasticsearch logs on either the production or marvel clusters.
Is this setup supposed to work? I haven't been able to find any documentation on the version requirements of elastic, marvel, and kibana when there is an external marvel cluster involved. If it is not supposed to work, in what order should I be performing the upgrades?