Waiting for Marvel Data after one sided 2.2.0 -> 2.3.1 upgrade

Elastic Stack Kibana
1

We're using an external marvel cluster to monitor our production elasticsearch cluster. As part of rolling out elasticsearch 2.3.1 I deployed it to the external marvel cluster but have not yet upgraded the production cluster. That has left us in a state where the production cluster is on 2.2.0 and the marvel cluster is on 2.3.1 with kibana 4.5.0.

At this point I'm only seeing the "Waiting for marvel data" screen when trying to use Marvel. I can confirm that there are marvel documents being indexed in the .marvel-es-2016.04.06 index as I can see them via kibana. There are also no errors in the elasticsearch logs on either the production or marvel clusters.

Is this setup supposed to work? I haven't been able to find any documentation on the version requirements of elastic, marvel, and kibana when there is an external marvel cluster involved. If it is not supposed to work, in what order should I be performing the upgrades?

Thanks!

2

Hi Mark,

Unfortunately, Marvel 2.3 has a backwards incompatibility related to how it handles the "source node" (aka how it determines if nodeX is the same nodeX that it saw earlier). As such, it creates a new versioned index called .marvel-es-1-* to avoid any issues with trying to merge the data between the old and the new.

This means that Kibana 4.5.0 plus its Marvel UI 2.3.x plugin is expecting the new index and not the old one. Since your production cluster has not yet been upgraded, it would be best to downgrade Kibana + Marvel UI plugin in the short term (assuming that upgrading your production cluster is not about to happen).

As you noticed, your production cluster is still happily publishing Marvel data to the monitoring cluster, using the old index format. Even though the monitoring cluster is running ES 2.3.1, it really comes down to the UI side (Kibana) that is querying it.

Officially, you are supposed to run Kibana 4.5 with ES 2.3, but I do not expect any issues with respect to Kibana 4.4 + Marvel UI 2.2 trying to read from the indices.

Moving forward, we have plans to make these types of upgrades more invisible and less coupled whenever possible.

1 Like
3

This worked out great. I rolled back to kibana 4.4 with marvel ui 2.2 and it came right up. I was able to leave the ES version in the marvel cluster at 2.3.1 and will upgrade all of this back to the latest versions when we roll 2.3.1 out to the production cluster.

Thanks!

1 Like