Translate Elasticsearch querries to Appsearch

Chenko · March 12, 2024, 4:31pm

Hello,

We have quite the situation on our hands.
We need to be able to translate the old ES queries to App Search. How would this be done?

Is this even possible for complexer queries?
We thought of using the /elasticsearch/_search endpoint in appsearch however this does not work with currations/synonyms/relevance tuning, making it useless for our use case.

One of the queries in question (One of the smaller ones):

{
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": [
            {
              "bool": {
                "must": [
                  {
                    "match": {
                      "metadata_44": {
                        "query": "verslag",
                        "operator": "and"
                      }
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "must": [
                  {
                    "term": {
                      "metadata_18.enum": "2019-2024"
                    }
                  }
                ]
              }
            },
            {
              "bool": {
                "should": [
                  {
                    "term": {
                      "metadata_25.enum": "Some text"
                    }
                  }
                ]
              }
            }
          ],
          "must_not": []
        }
      },
      "functions": [
        {
          "weight": 0,
          "filter": {
            "terms": {
              "string": [
                "string"
              ]
            }
          }
        },
        {
          "script_score": {
            "script": {
              "source": "COMPLEX SCRIPT"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "X_TYPE": {
      "terms": {
        "field": "metadata_25.enum",
        "size": 1000,
        "order": {
          "_count": "desc"
        }
      }
    },
    "X_AG": {
      "terms": {
        "field": "metadata_1.enum",
        "size": 1000,
        "order": {
          "_count": "desc"
        }
      }
    },
    "X_AG_TYPE": {
      "terms": {
        "field": "metadata_2.enum",
        "size": 1000,
        "order": {
          "_count": "desc"
        }
      }
    },
    "X_I": {
      "terms": {
        "field": "metadata_16.enum",
        "size": 1000,
        "order": {
          "_count": "desc"
        }
      }
    },
    "X_L": {
      "terms": {
        "field": "metadata_18.enum",
        "size": 1000,
        "order": {
          "_key": "asc"
        }
      }
    },
    "X_Z": {
      "terms": {
        "field": "metadata_48.enum",
        "size": 1000,
        "order": {
          "_key": "asc"
        }
      }
    },
    "X_STATUS": {
      "terms": {
        "field": "metadata_29.enum",
        "size": 1000,
        "order": {
          "_count": "desc"
        }
      }
    },
    "X_IN": {
      "terms": {
        "field": "metadata_14.enum",
        "size": 1000,
        "order": {
          "_key": "asc"
        }
      }
    },
    "X_VR": {
      "terms": {
        "field": "metadata_47.enum",
        "size": 1000,
        "order": {
          "_key": "asc"
        }
      }
    },
    "X_MI": {
      "terms": {
        "field": "metadata_20.enum",
        "size": 1000,
        "order": {
          "_key": "asc"
        }
      }
    },
    "X_INV": {
      "terms": {
        "field": "metadata_17.enum",
        "size": 1000,
        "order": {
          "_key": "asc"
        }
      }
    },
    "X_CO": {
      "terms": {
        "field": "metadata_8.enum",
        "size": 1000,
        "order": {
          "_count": "desc"
        }
      }
    },
    "X_THM": {
      "terms": {
        "field": "metadata_42.enum",
        "size": 1000,
        "order": {
          "_count": "desc"
        }
      }
    },
    "X_SOR": {
      "terms": {
        "field": "metadata_28.enum",
        "size": 1000,
        "order": {
          "_count": "desc"
        }
      }
    },
    "X_YS": {
      "terms": {
        "field": "metadata_jaar.enum",
        "size": 1000,
        "order": {
          "_key": "asc"
        }
      }
    }
  }
}

So our situation is that we need to use app search for this, is this possible?

Again my questions:

Is it possible to translate queries like this one to App Search?
Is it possible to get the same results/ results that fit the query ?
How would I translate this query to app search, and future querries?

Kathleen_DeRusso · March 12, 2024, 4:44pm

We thought of using the /elasticsearch/_search endpoint in appsearch however this does not work with currations/synonyms/relevance tuning, making it useless for our use case.

Have you checked out query rules and synonyms APIs? I suspect you could get pretty far using Elasticsearch's main _search endpoint with these features. You'd have to adjust the weights for relevance tuning manually but it would give you more control over your Elasticsearch queries.

Sean_Story · March 12, 2024, 4:53pm

Hi @Chenko ,

First, it's worth stressing that App Search is intended simpler than Elasticsearch on the scale of Simplicity <--> Power/Customizability. This means that App Search strictly cannot do everything that Elasticsearch can do, unless you choose to use the Elasticsearch Search API. If you need the low-level control of Elasticsearch's APIs, you might be better served using Elasticsearch instead of App Search Engines.

For the specific query you shared, this is not so complex that you can't get something very similar with App Search.

Pieces like:

{
  "term": {
    "metadata_25.enum": "Some text"
  }
}

can be replaced with App Search Value Filters.

Pieces like

{
  "match": {
    "metadata_44": {
      "query": "verslag",
      "operator": "and"
    }
  }
}

can be replaced with the query parameter.

Aggs like

"terms": {
  "field": "metadata_25.enum",
  "size": 1000,
  "order": {
    "_count": "desc"
  }
}

can be replaced by specifying Value Facets. However, note that the max size for a value facet is 250, and your query has 1000 set. So you'll get fewer buckets if you have more than 250 distinct values in these fields.

The only piece in your example that won't directly translate is:

"functions": [
  {
    "weight": 0,
    "filter": {
      "terms": {
        "string": [
          "string"
        ]
      }
    }
  },
  {
    "script_score": {
      "script": {
        "source": "COMPLEX SCRIPT"
      }
    }
  }
]

Your options here are to either look at the App Search Boosts and try to translate COMPLEX SCRIPT into something there, or you could use the Search Explain API to get the Elasticsearch syntax for the App Search query you've 90% formed up, and then tweak that Elasticsearch query to use your COMPLEX SCRIPT and then issue the modified Elasticsearch query with the Elasticsearch search API.

Hope this helps. Good luck!

Chenko · March 12, 2024, 4:59pm

Would I have access to the synonyms and currations defined in app search?
When normally using the /elasticsearch/_search api I do not have access to those, is there a way to get them?

Chenko · March 12, 2024, 5:06pm

Thanks for the large response! My thoughts were also to use the search_explain api to get the "raw" Elasticsearch query however I would think this would be too much of a mess to maintain.

Kathleen_DeRusso · March 12, 2024, 5:58pm

No, you'd have to use the Elasticsearch regular _search endpoint, but you could do this against the App search backed Elastic indices if you needed to.

Chenko · March 12, 2024, 6:14pm

So we stil would not be able to use the synonyms/curations we set with App Search?

If so I do not think this is an option we can take unfortunately.

Sean_Story · March 12, 2024, 8:52pm

You can't have your cake and eat it too, I'm afraid. Like I'd said before, you either need to choose the "simplicity" of App Search, and acknowledge that means less fine-grained control, OR you choose the "power" of Elasticsearch, and acknowledge that means more complexity to maintain.

Chenko · March 12, 2024, 9:25pm

Okay so what you're saying is that it is simply "impossible" to recreate the Elastic Search queries in App Search, while enjoying the simplicity of App Search?

Sean_Story · March 13, 2024, 2:00pm

App Search can only express a subset of Elasticsearch's full capabilities.

Everything App Search does can be done directly in Elasticsearch (it's using Elasticsearch under the hood, and the Explain API tells you exactly what its doing). So if you want to set up things like synonyms and curations in App Search, but then use a feature that App Search doesn't support (like script scores), you'll need to convert the App Search query to an Elasticsearch query, and modify it to meet your needs.

Alternatively, you can choose to use Elasticsearch-native features like Query Rules and the Synonyms APIs, like @Kathleen_DeRusso had originally suggested.

You have a number of options.

Chenko · March 13, 2024, 2:20pm

I see Thank you both for the responses!

system · April 10, 2024, 2:21pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.