Elasticsearch version and plugin compatibility

Elastic Stack Elasticsearch
1

What is the compatibility between a certain elasticsearch version and plugins made for similar versions? A specific example - let's say I have a plugin made for elasticsearch 2.3.1 installed and working. Then there is a minor bug fix for elasticsearch and they bump the version to 2.3.2. Will the plugin made for 2.3.1 still work with the new version? Is it possible for the plugin developer say their plugin works with 2.3.x or it has to be compiled for each minor version level.

Thanks

2

I just found https://www.elastic.co/guide/en/elasticsearch/plugins/2.3/plugin-authors.html

Plugin release lifecycle
You will have to release a new version of the plugin for each new elasticsearch release. This version is checked when the plugin is loaded so Elasticsearch will refuse to start in the presence of plugins with the incorrect elasticsearch.version.

Wow.. This is really terrible. Please someone tell me it's not true.

Why can't there be an interface that the plugin talks to and as long as the interface doesn't change, the plugins should still work. Eg, I have a set of plugins for different things, and there is a bug fix in elasticsearch - why should I have to update my other plugins?

I can't imagine how you develop a strong plugin ecosystem around an approach that's so fragile, especially for something that's moving as fast as elasticsearch.

3

It is true.

To make sure it actually works (and not just should work), you have to run the tests again and release the new version.

The bug fix might introduce a functional change - I mean the behavior behind the API might change - and even if the API would still the same your plugin can introduce another issue.

As a plugin author as well, I hear you. It could be annoying.
That being said, I've seen in the past when I was working on analysis plugins that if you are providing for example a plugin which does not use the same Lucene version behind the scene, it would stop your node from starting at best or could cause bad issues for your existing indices.

My opinion is that It's definitely better to be safe for the end user even it's annoying for the plugin developer.
Elasticsearch is storing / indexing a lot of data and potentially critical mission ones. We took the decision to take this seriously and not take any risk.

I'm sorry for you (and for other plugin developers - like me) but this is the price to pay.

4

Then it's not a bug fix. This is why programming systems have interfaces - to have a contract on the interaction. Sometimes something requires a fundamental change and the contract needs to be changed. Eg, It use to be that request from the response would return a json array and now we want it to return an object with metadata about the response and the data will now be in a "results" property - that's a breaking change and unless plugins are built for the new structure they will at best not work and at worse make ES unstable/broken.

But there are other times where there's a "bug" - we had a contract on what would happen, but I'm not doing it right. I allowed two sort fields - sort[0]=time:desc; sort[1]=score:asc - and I accidentally reversed the order in my code so I did 1 and then 0. When I fix that, it shouldn't affect anyone.

I would be interested you could look at Issues · elastic/elasticsearch · GitHub and find an example that would break a plugin.
The one exception is where you had agreed on an interface, had implemented it wrong, plugin authors worked with the broken interface - now that that bug is fixed, their code will break.

It seems the correct approach is to have an API/plugin interface version which would get updated when the internals change.

I'm not a plugin author, but I understand better why there are so many out of date/abandoned plugins.

So, a lucene version bump can be a significant change and I understand how that could be a problem. But according the to release notes looking at 2.0.x, 2.1.x, 2.2.x and 2.3.x, minor releases never included lucene upgrades.

So it's nice that you care about my data, but I have plenty of ways to destroy all my data without your help.
DELETE * - does a pretty nice job. So why don't you take that away also? I think what would be more helpful is that you provide an override option like with the delete action.destructive_requires_name:true (except that there the default is the dangerous option). You could allow people to run all or some plugins that are out of date/version and let me decide on the risk based on what the plugin is doing. The default setting could and should be safe, but for people that know what they are doing, they could and should be allow to override it, IMO.

I'm just not convinced that the only possible solution is to be heavy handed like this. I would be interested to see how other servers with robust plugin communities handle this.