Free text search with App Search/Elastic Search UI with string , AND/ OR options?

[thomas-ld (Thomas)]

Does Elastic AppSearch/UISearch support implementing free text search on large text fields ?

I would like to use ElasticSearchUI / SearchBox with option to search like:

"whale was seen in" + houston - dalas

I.e. use " " for exact match of string and +/- (or AND/OR) to include exclude other terms.

Is that possible by using Elastic AppSearch/UI search ? I'm using the react example app generated from Elasticsearch as base for my application, behavior of the provided SearchBox seems to be single word terms with OR as default between. I haven't found any config parameters to change that behavior, is it even possible?

Any tips/ideas are appreciated!

[thomas-ld (Thomas)]

Ok, I now see in the browser debugger that the query string looks like this when passed on to elastic app search: ""whale was seen in" + houston - dalas"

Is it perhaps just a matter of enabling Lucene Query Syntax in the App Search engine? (Or when creating the index ?) How to do that ?

[Carlos_D (Carlos)]

Hi @thomas-ld !

Lucene query syntax is supported (see docs).

The following test works on the test engine (national-parks-demo) using Search UI: "situated on" -olympic +yellowstone

The default behaviour when not using Lucene syntax is to try finding as many terms as possible, depending on the relevance of the terms. If you need to find specific terms, I'd suggest you use the Lucene syntax.

Is there anything else we can help you with?

[Kathleen_DeRusso]

Hi @thomas-ld ,

Unfortunately there is a known issue with App search that Lucene boolean search syntax can return unexpected results.

Our recommended workaround here is to use the Elasticsearch Search API for App Search, which will allow you to send Elasticsearch requests (such as Boolean queries) to your App Search indices.

This will send in an Elasticsearch request that overrides app search’s configuration.If you use the explain API, though, you can see the queries that are being passed in and you can adjust them. The issue with lucene syntax today involves the minimum_should_match value, which often conflicts with Lucene’s boolean syntax logic.If you’re doing an AND search with a minimum_should_match of 1, it effectively becomes an OR query (only one needs to match).So it’s a bit of a workaround, but using explain to get the query you want and transforming it to Elasticsearch query DSL is the best way to have complete control over your result set.

[thomas-ld (Thomas)]

Ok, thank for the feedback!
I think my problem is my index, I have created the index with elastic directly and not via app search.
When looking at the index in app search i see that message: "Missing subfields" for my text fields I'm searching in. Reading the docs I see that there are some subfields required for appsearch to work properly.
Ref:

Elasticsearch engines text field conventions | App Search documentation [8.10]...

I then tried to create a new index with a field "text" with subfields to try to mimic what is said in the link above (see mapping below). But still get same warning. Do you know how to create the subfields to get a field properly working with app search ? Should type be something else than text for the different sub fields ?

This is how I created the index:

{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "properties": {
      "Date": {
        "type": "date"
      },
      "text": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          },
          "category": {
            "type": "text"
          },
          "prefix": {
            "type": "text"
          }, 
          "delimiter": {
            "type": "text"
          },
          "joined": {
            "type": "text"
          },
          "stem":{
            "type": "text"
          }
        }
      }	
    }
  }
}

[Kathleen_DeRusso]

text should be fine, you can find out more information on how to customize these mappings in our text field support conventions documentation.

Here is an example of the default mappings that App search creates when an index is created:

{
  ".ent-search-engine-documents-test": {
    "mappings": {
      "dynamic": "true",
      "dynamic_templates": [
        {
          "permissions": {
            "match": "_*_permissions",
            "mapping": {
              "type": "keyword"
            }
          }
        },
        {
          "thumbnails": {
            "match": "_thumbnail_*",
            "mapping": {
              "type": "binary"
            }
          }
        },
        {
          "data": {
            "match_mapping_type": "*",
            "mapping": {
              "analyzer": "iq_text_base",
              "fields": {
                "date": {
                  "format": "strict_date_time||strict_date",
                  "ignore_malformed": true,
                  "type": "date"
                },
                "prefix": {
                  "search_analyzer": "q_prefix",
                  "analyzer": "i_prefix",
                  "type": "text",
                  "index_options": "docs"
                },
                "delimiter": {
                  "analyzer": "iq_text_delimiter",
                  "type": "text",
                  "index_options": "freqs"
                },
                "joined": {
                  "search_analyzer": "q_text_bigram",
                  "analyzer": "i_text_bigram",
                  "type": "text",
                  "index_options": "freqs"
                },
                "location": {
                  "ignore_malformed": true,
                  "type": "geo_point",
                  "ignore_z_value": false
                },
                "float": {
                  "ignore_malformed": true,
                  "type": "double"
                },
                "enum": {
                  "ignore_above": 2048,
                  "type": "keyword"
                },
                "stem": {
                  "analyzer": "iq_text_stem",
                  "type": "text"
                }
              },
              "index_options": "freqs",
              "type": "text"
            }
          }
        }
      ],
      "properties": {
        "id": {
          "type": "keyword"
        }
      }
    }
  }
}

[thomas-ld (Thomas)]

@Kathleen_DeRusso

analyzer

Now created a new index with following subfields added to the text fields and then string and exclusive search works as expected! Thanks for the support!

		"prefix": {
		  "type": "text",
		  "index_options": "docs"
		},
		"delimiter": {
		  "type": "text",
		  "index_options": "freqs"
		},
		"joined": {
		  "type": "text",
		  "index_options": "freqs"
		},             
		"stem": {
		  "type": "text"
		}