Moving From 1.7.5 to 7.10.2 - DLS Changes

Elastic Stack Elasticsearch
1

I've used an ancient (1.7.5) version of Elasticsearch for years and it's been great. The initial setup was a bit difficult, but when completed it has always been rock solid.

Unfortunately my Elasticsearch provider wants me to upgrade or pay an exorbitant amount to keep using 1.7.5.

So, here's an example of what I have to upgrade, which works perfectly in 1.7.5 but gives an "Unknown key for a START_OBJECT in [filter]." error.

Basically I'm trying to figure out why this is happening and what changes to the DSL am I missing in 7.10.2 to trigger this error.

Any help would be much appreciated.

The query:

{
		query: {
		multi_match: {
			query: q, fields: ['title^10', 'body']
		},

		},
		filter: {
		bool:{
			must: [
				{term: {root_comment: true}},
				{term: {soft_delete: false}},
				{term: {is_draft: false}},
				{term: {"sub.is_private": {value: false}}},
				{term: {"sub.is_banned": {value: false}}},
				{term: {"sub.nsfw": false}}
			]
		}	
	},
	"sort": [
		{ "posted": { "order": "desc" }},
		{ "_score": { "order": "desc" }}
	],
	from: start,
	size: limit
}
2

As you are making such a large leap in versions I do not see the point in going to a version that is also very old and have been EOL a long time. I would recommend going all the way to the very latest Elasticsearch 8 release as that likely will save you a lot of time and effort as version 7.10 may be dropped soon too.

3

I hear you, but basically the last Open Source version is 7.10.2 and it's the difference between $9/mo and $30.

Basically I'm on Heroku and it's an add on...which is kind of the whole problem.

4

What is the mapping for the fields you are querying? Looks like Boolean, but would like to make sure. It would probably help if you could create a minimal complete example that highlights the issue and can be run in Kibana console.

5

Why are these different? Should this not be:

6

This is a Ruby on Rails app, so I'm just using the elasticsearch, elasticsearch-model, and elasticsearch-rails GEMs. Therefore I've never had to map anything - it just worked. Basically the all of the fields are properties of Ruby objects that Rails then saves in Elastic. So, "title", "body", "root_comment", etc. are all columns in postgres in the "comments" table, which get saved to Elastic when a comment is created.

7

I am not a Ruby developer so am not familiar with that. I can only help if the example is in API calls instead of using client libraries.

8

Sorry, yeah, you're right. But see my other comment - this is coming from a Rails app, so the code may have remnants of that. The extra } is from that...

9

Then you will need to wait for someone familiar with that client library to respond. Have you upgraded the client library to a version compatible with Elasticsearch 7.10?

10

Yes, I'm running it locally and also with the provider, both on 7.10.2.

So I can use curl locally with just basic Elasticsearch queries, like the one I posted.

11

Then revise the query as per my comments and try it through curl and see if that resolves the issue or at least gives a different error message.

12

Yeah I have. Basically it returns the entire index and then says:

query:: command not found
multi_match:: command not found
query:: command not found
},: command not found
},: command not found
bool:{: command not found
Command 'filter:' not found, did you mean:
  command 'filter' from deb filter (2.6.3+ds1-3)
  command 'filterm' from deb konwert (1.8-13.2)
Try: sudo apt install <deb name>
{term:: command not found
{term:: command not found
{term:: command not found
{term:: command not found
{term:: command not found
{term:: command not found
},: command not found

I have rewritten it and it "worked" in that it returned search both in curl and in the app, but the results were wrong due to the order. Basically using bool, then must, and then the multi_match.

What I'm saying here is I think, can't be sure, that the DSL was changed and I don't understand where.

13

Here's the example that I used for the basis of my query that "worked".

Unfortunately I blew my version away, but the point is the order - while it works it doesn't do the same logic.

def self.search(query, genre = nil)
    params = {
      query: {
        bool: {
          must: [
            {
              multi_match: {
                query: query, 
                fields: [ :title, :artist, :lyrics ] 
              }
            },
          ],
          filter: [
            {
              term: { genre: genre }
            }
          ]
        }
      }
    }

    self.__elasticsearch__.search(params)
  end
14

Here's the query that "works" but doesn't have the same logic.

query: {
		bool:{
			must: [
				{

		multi_match: {
			query: q, fields: ['title^10', 'body']
		}
				},

				],
		filter: [
				{term: {root_comment: true}},
				{term: {soft_delete: false}},
				{term: {is_draft: false}},
				{term: {"sub.is_private": {value: false}}},
				{term: {"sub.is_banned": {value: false}}},
				{term: {"sub.nsfw": false}}
		]
	}
	},
	"sort": [
		{ "posted": { "order": "desc" }},
		{ "_score": { "order": "desc" }}
	],
	from: start,
	size: limit
}