Elasticsearch-rails & -ruby are abandoned

Hello,

this thread escalated faster than I had a chance to react, unfortunately.

I would like to return to the original question, and focus us back on the very important topic: the development and maintenance of the Ruby client and the Rails integrations for Elasticsearch.

The first crucial aspect to address is that the Ruby client and the Rails integrations are two completely separate projects, which together contain eight different Rubygems (not counting the X-Pack integration, with a very different degree of problematic aspects. For any further discussion, let's keep this distintion in mind.

The Ruby client has 6 open pull requests right now, with [5 pull requests] waiting for reply from the contributor. There are also 10 issues with questions waiting for reply, and 26 issues with general discussion waiting for reply. The client and associated libraries, like elasticsearch-extensions are consistently updated, with a lot of activity from October 2017, actively tracking the changes in the Elasticsearch 6.x branch. Travis CI currently fails because of incorrect snapshot URL for the Elasticsearch 6.0 package, not with a load of actual test failures. That, to me, doesn't look like an "abandoned" project at all. If it does to our community, I'm very much in favour of a discussion about how can we improve here.

The Rails integration is a different story, and I understand the frustration expressed in the original post by @kshnurov. The project itself consists of three totally different Rubygems: the integration with ActiveRecord models, the persistence layer, and the Rails helpers and extensions. All three are pretty different when it comes to complexity, usage — and community contibutions. Just compare the download numbers between elasticsearch-model and elasticsearch-persistence: 3 million versus 600 thousands. But one of biggest sources of troubles, manifesting in the number of issues and pull requests submitted, is actually elasticsearch-persistence, which skews the perspective quite significantly.

With that in mind, let's consider the technical details. For a good example, I've been actively and publicly telling people that the ActiveRecord pattern for the elasticsearch-persistence is too challenging from the development and usage standpoint, and that it's going to be deprecated in favour of the Repository pattern. Now, you can hold it against the maintainer, in this case me, to not act on such a plan in a very long time. Doing this transition properly, which means updating the code with deprecation notices, updating the Rails example application so people are not left out in the cold with a "deprecation notice", writing a blog post about this transition and the motivation, would help the community very much, and I am personally frustrated that it didn't happen.

Another good example are the pull requests themselves. As is the case with any larger open source project, there's a tension between the stability of the library and the agility with updating the code. This is very important to keep in mind.

In optimal cases, the project strikes a good balance, and the Rubygems for the Rails integration current do not achieve this. The reasons for that are predictable, no matter how much frustrating they can be: merging a contribution is not as easy as clicking the button at Github. For many pull requests — and here I want to repeat what I've said on a similar thread couple of months ago — the first complication is that it's not evident what is actually the real problem, and if the solution is in a correct part of the source code. This is amplified by the pull request not having tests, for example. The burden is than on the maintainer to: understand the problem at hand, verify the correctnes of the solution, provide code-based evidence for the problem in a unit/integration test, rework the entire contribution, and push it to the repository. As you might imagine, you may easily spend an hour or two doing that, and that significantly impacts the agility aspect mentioned above.

In a situation like that, a maintainer — me, in this case — can make a conscious decision to focus more on the stability aspect, in order to not break people's applications. Granted, many people and contributors will be frustrated by the lack of said agility, but a much more greater number of people would be impacted by careless, "agile" approach to updating the codebase.

With all that said, we are still actively trying to find a good solution here, in order to be more agile with the development and maintenance of the Rubygems for Rails integration. Of course, you may wonder, what's so hard about that for a company with $104m of funding, that's just silly, no? But in reality, it is surprisingly hard.

Returning back to you original question: "are the Rubygems for Elasticsearch abandoned?", the short answer is "no, not at all".

The larger answer is that in a tight situation, we made a conscious decision to keep focus on the Ruby client, which is the foundation for any other Ruby integration and application, and to actively search for a solution for developing and maintaining the Rubygems for Rails integration without breaking existing applications.

In this thread, I have tried to provide an honest answer to the issue raised — I do realize it's quite long, hopefuly it's not overwhelming, and will at least a little bit assure you that what you see is not just mere negligence and that the situation is a bit more complex than it seems from a first sight.

Best,

Karel

9 Likes