Curious why for each dot release of elasticsearch (latest was 2.3.4 to 2.3.5) I have to uninstall the license and marvel-agent plugins then reinstall them? Why aren't they just bundled with the distribution of elasticsearch or somehow automatically updated with the new version of ES? Will this be "fixed" with the 5.0 release maybe?
Hi Jerry,
The reason is honestly more pragmatic than technical: we used to support many versions of Elasticsearch with a single version (e.g., the latest version of Marvel 1.x supported many ES 1.y releases), but what we found is that it became a nightmare to operationally maintain the matrix for understanding "plugin X works with ES versions A, B, C, but not D". Even as an expert on the subject, I constantly found myself double-checking "does XYZ v1.2.3 work with ES 1.y" and, worse, we found users accidentally upgrading with then-invalid plugins still installed. Now there's no question in either scenario.
By pivoting to lock-step versioning with Elasticsearch itself, we are able to do a bunch of things: guarantee that the code statically works with each version (we naturally have to recompile it each time), as well as that all integration tests were run against the version being released. Also of interest, we also benefit from Elasticsearch because each plugin adds metadata about what version it was compiled against, so it can proactively avoid running "invalid" plugins, thus giving a better opportunity for a smoother experience (as opposed to unknown-unknowns).
The only downside is that you, as the operator, need to reinstall it with each release. Ideally you have scripts installing everything, but we recognize that many people don't do that and we're sorry that it does add a little bit of nuisance.
Why aren't they just bundled with the distribution of elasticsearch or somehow automatically updated with the new version of ES?
Elasticsearch represents the open source compilation, and nothing more.
Will this be "fixed" with the 5.0 release maybe?
It depends on your definition of "fixed". We are improving our approach in Elasticsearch (and Kibana!) 5.0. We will still require lock-step versioning, but we will be packaging Shield, Watcher, License, Marvel Agent, Security UI, and Marvel UI (!) as a single plugin: X-Pack.
So, you'll still need to uninstall and reinstall, but you'll literally use the same file for all them and just once per application (x-pack needs to be installed in both ES and Kibana, but the same file gets used and each application grabs what it needs; if you connect online then all that matters is that it's the same name).
Hope that helps,
Chris