@stephenb we've gotten to point of trying this out now. We've seen that using NEST 7.x against an ES 6.x cluster will work for search operations () but not for indexing (). The error we get there, no surprise, "no type was specified". Of course, we need indexing to also continue to work for our application.
So, one last ditch question to see if we can avoid an outage in order to roll the cluster upgrade and then deploy the new code:
Could you see if someone on the NEST team could tell me if there is there some compatibility mode available or some strategy we could use to deploy NEST 7.x code while still on ES 6.x? Perhaps some C# extension method for .Type() that could detect the version of ES or some other clever approach to give us a window to avoid an outage?
I will take a quick poke and see if I can... but I would not expect anything..
6.x to 7.x was alway considered "A Breaking Change" ..
You could look at the code directly in the repo... for the NEST client it is all open.
My first instinct / (admittedly not well though through) is to add a a default pipeline to your indices so that you can add the appropriate _type
index.default_pipeline
The default ingest node pipeline for this index. Index requests will fail if the default pipeline is set and the pipeline does not exist. The default may be overridden using the pipeline parameter. The special pipeline name _none indicates no ingest pipeline should be run.
Thanks @stephenb, we've got a pretty simple deployment and have never used Ingest nodes but we're taking a look at that option. Seems like a promising possibility.
You don't need dedicated ingest nodes to use ingest pipelines.
Unless you have specifically set up nodes to not be ingest nodes they should run ingest pipelines.
Try it. You can create a simple little pipeline and just execute a test.
You wanted ideas. That's my idea... At this point .... Not saying it's the best idea but it's the one I got right now
Unfortunately, there is no compatibility mode for 6x to 7x, and the upgrade is likely to require a short downtime to switch. If you're able to run a 6x and 7x side-by-side and sync the data, you may be able to take advantage of our namespaced packages to use a 6x and 7x client instance in the same app, but that's often more complex than a timed upgrade of both client and server.
Note that we introduced a compatibility mode for 8x to make client upgrades easier so when you're ready for that upgrade, things are smoother.
@stephenb@stevejgordon thank you both for your replies. We're investigating ingest pipelines as a potential "no downtime" option though, even if that works for indexing, we'll have to do broad testing to verify that all our other use cases also work (ie. snapshots, rollovers, alias creation, etc.). In the end, some downtime may be the safest, most straightforward way to go.
We're definitely looking forward to being able to run NEST 7.x on an ES 8.x cluster. We haven't started the investigation there but we're hoping that's reasonably straightforward. Thanks again for the responses!
@stephenb@stevejgordon turns out that there is a way to get NEST 7.x to play nice with ES 6.x! As I said previously, search works just fine. For indexing, one of our engineers discovered TypeQueryString on the bulk descriptor which satisfies ES 6. This puts the type as an HTTP query parameter and it works against both ES 6 and ES 7. We've never used that in the past so I can't speak to it's original intended purpose but, in the event someone else inquires about this, it seems to be working for us across the board. That is, all the "ancillary" functions we use are working as well (ie. alias CRUD, rollover, snapshot, index CRUD).
The only footnote so far is that the rollover command succeeds but the NEST 7 library claims it was not a valid response. We can live with this as we roll the cluster upgrades behind the code upgrade but, assuming we don't run into anything else, this will allow us to have zero downtime.
I don't know if this use of TypeQueryString was intended or is just a happy accident but if it's the former, I'd expect it to be more common knowledge as a compatibility mode during transition.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
) but not for indexing (
). The error we get there, no surprise, "no type was specified". Of course, we need indexing to also continue to work for our application.