Text-search using URI param query failing after upgrade to 6.2.2 from 5.4.1

I have a mapping for a field Tenant defined as follows.
"Tenant": {
"type": "keyword"
},

And I use to do elastic query using URI param
localhost:9200/_search?q=foo (simple text search on "foo")
and used to get matching docs with Tenant set "foo" with elastic 5.4.1

After upgrading to elastic 6.2.2, the same query with the same index mapping is not returning any matching result.
{"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":0,"max_score":nul

Has something changed in 6.x.x that requires different index mapping or query format for URI param for text search?

Do I need to have both text and keyword mapping defined - for fields that needs text search as well as aggregation support with 6.x.x ?

Nothing that would affect an URI search has changed as far as I know. I reproduced your situation and was successful. Do you have document in your cluster that match "foo"?

I see the text search is failing for nested objects only with the same schema that worked with 5.4.1

  "mappings" : {
      "configs" : {
        "properties" : {
          "APIVersion" : {
            "type" : "text"
          },
          "Kind" : "TypeFoo",
          "meta" : {                <<<<<
            "type" : "nested",
            "properties" : {
              "CreationTime" : {
                "type" : "date"
              },
              "ModTime" : {
                "type" : "date"
              },
              "Name" : {
                "type" : "text"
              },
              "Tenant" : {
                "type" : "keyword" <<<<
              },

Here is the object in Elastic.

        "_source" : {
          "APIVersion" : "v1",
          "Kind" : "TypeFoo",
          "meta" : {
            "Name" : "sg03",
            "Tenant" : "audi",  <----
...
           }

The following text query returns no match with 6.2.2 but works with 5.4.1

curl localhost:9200/myindex.*/_search?q=audi
{"took":2,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}

Also also these term query on nested object fails with no match.

    curl localhost:9200/myindex.*/_search?q=meta.Tenant:audi

    curl localhost:9200/myindex.*/_search?q=Tenant:audi

This query on outer object works. (text or term)

    curl localhost:9200/myindex.*/_search?q=TypeFoo

    curl localhost:9200/myindex.*/_search?q=Kind:TypeFoo

That's because you can't do nested queries on nested objects without using the QueryDSL I think.

Thanks for the response.

Text search using URI param is working with 5.4.1 (text search at any object level) and did it change with 5.5 or in 6.x ?

         curl localhost:9200/myindex.*/_search?q=audi.    <<<<<<

Is there a workaround to get text search working with 6.2.2 for any object level using URI query string parameter ?

Are you sure the mapping in 5.x had this nested type in it?

Can you run:

GET your_index/_mapping

On the 5.6 cluster?

Yes. The mapping is same on both versions and Tenant is nested attribute.

Here is the output with 5.4.1.

It is very easy to repro.

 $ curl localhost:9200
{
  "name" : "ZgsXWKs",
  "cluster_uuid" : "z8Ev1YWOTESRztEivh73HQ",
  "version" : {
    "number" : "5.4.1",       <<<<<<<<<<<<<<<<
    "build_hash" : "2cfe0df",
    "build_date" : "2017-05-29T16:05:51.443Z",
    "build_snapshot" : false,
    "lucene_version" : "6.5.1"
  },
  "tagline" : "You Know, for Search"
}

$ curl localhost:9200/myindex.*/_mapping?pretty
{
    "mappings" : {
      "configs" : {
        "properties" : {
          "APIVersion" : {
            "type" : "text"
          },
          "Kind" : {
            "type" : "keyword"
          },
          "meta" : {
            "type" : "nested",  <<<<<<<<<<<<<<
            "properties" : {
              "CreationTime" : {
                "type" : "date"
              },
              "Name" : {
                "type" : "text"
              },
              "Namespace" : {
                "type" : "keyword"
              },
              "Tenant" : {
                "type" : "keyword".  <<<<<<<<<<<<<
              },
            }
          },

Can you share a full repro script that I can reuse?

Sorry we dont have a standalone script. But direct-messaged you the index mapping . Appreciate your help in this regard.

Another data point - URI text search on nested objects for the mapping I had works in 5.5.0 as well.

It starts failing from 6.0.0 onwards.

So if I have time to create a reproduction script, I'll look at it.
Unless someone else has spare time.

Why not sharing that here?

It is in GO backend code with properietary code inter-twined and would take more effort share it in public.

Maybe will provide CURL and bash scripts some time this week.

Here are steps to repro on 6.2.2

# verify elastic endpoint is up with version 6.2.2
curl localhost:9200

# create myindex with mapping
curl -H 'Content-Type: application/json' -XPUT -d '{ "mappings": { "configs": { "properties": { "Kind": { "type": "keyword" }, "meta": { "type": "nested", "properties": { "Name": { "type": "text" }, "Tenant": { "type": "keyword" } } } } } } }' http://localhost:9200/myindex

# verify mapping
curl localhost:9200/myindex/_mapping?pretty

# add doc/object
curl -H 'Content-Type: application/json' -d '{ "Kind": "foo", "meta": { "Name": "sg03", "Tenant": "audi" } }' http://localhost:9200/myindex/configs

# verify object
curl localhost:9200/myindex/_search?pretty

# text search on outer field (Kind's) value
curl localhost:9200/myindex/_search?q=foo

# text search on nested field (Tenant's) value
curl localhost:9200/myindex/_search?q=audi

Here is my repro with output shown ...

MacBook-Pro[search] $ curl localhost:9200
{
  "name" : "iP_9TW_",
  "cluster_name" : "test-elasticcluster",
  "cluster_uuid" : "HtH7CfY0Sd6QxoAtBaxVUA",
  "version" : {
    "number" : "6.2.2",
    "build_hash" : "10b1edd",
    "build_date" : "2018-02-16T19:01:30.685723Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

MacBook-Pro[search] $  curl -H 'Content-Type: application/json' -XPUT -d '{ "mappings": { "configs": { "properties": { "Kind": { "type": "keyword" }, "meta": { "type": "nested", "properties": { "Name": { "type": "text" }, "Tenant": { "type": "keyword" } } } } } } }' http://localhost:9200/myindex
{"acknowledged":true,"shards_acknowledged":true,"index":"myindex"}

MacBook-Pro[search] $ curl localhost:9200/myindex/_mapping?pretty
{
  "myindex" : {
    "mappings" : {
      "configs" : {
        "properties" : {
          "Kind" : {
            "type" : "keyword"
          },
          "meta" : {
            "type" : "nested",
            "properties" : {
              "Name" : {
                "type" : "text"
              },
              "Tenant" : {
                "type" : "keyword"
              }
            }
          }
        }
      }
    }
  }
}

MacBook-Pro[search] $ curl -H 'Content-Type: application/json' -d '{ "Kind": "foo", "meta": { "Name": "sg03", "Tenant": "audi" } }' http://localhost:9200/myindex/configs
{"_index":"myindex","_type":"configs","_id":"KpGcIGIBHBDa29EzJsNU","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

MacBook-Pro[search] $ curl localhost:9200/myindex/_search?pretty
{
  "took" : 85,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "myindex",
        "_type" : "configs",
        "_id" : "KpGcIGIBHBDa29EzJsNU",
        "_score" : 1.0,
        "_source" : {
          "Kind" : "foo",
          "meta" : {
            "Name" : "sg03",
            "Tenant" : "audi"
          }
        }
      }
    ]
  }
}

## Text search on outer field - HIT/WORKS
MacBook-Pro[search] $ curl localhost:9200/myindex/_search?q=foo
{"took":18,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":1,"max_score":0.2876821,"hits":[{"_index":"myindex","_type":"configs","_id":"KpGcIGIBHBDa29EzJsNU","_score":0.2876821,"_source":{ "Kind": "foo", "meta": { "Name": "sg03", "Tenant": "audi" } }}]}}

## Text search on inner field - MISS
MacBook-Pro[search] $ curl localhost:9200/myindex/_search?q=audi
{"took":2,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}

Elastic team - Please let me know if you need more info with above mentioned repro steps.

It is blocking our upgrade to 6.2.2 and really appreciate your time and help on the QueryString style query - text search on nested objects. We see this issue in all request forms - CURL with URI param, GET with body or even in code using elastic go client (querystring query).

I think that the difference is that we don't generate anymore this catch all field _all anymore.

I believe that if you really need to do that, you have to use the copy_to feature and generate a _all like field which you can use in the query string query as the "default field".

I see.

May I know how do we make it work for dynamic mappings?

We have defined static mapping for meta/header fields in objects and rest of the fields is dynamically mapped. My requirement is to provide a simple text search service on all object fields, including nested - either coming via static mapping pre-defined or dynamic mapping.

You can use this: https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html

Thanks David for the help.

Have a final question for this thread.

I tried the copy_to and it works for first level nested object, but doesnt work for second and further level nested objects. May I know how to make it work for the meta.Labels.key field to use copy_to in this example

{
	"mappings": {
		"configs": {
			"properties": {
				"catch_all": {
					"type": "text"
				},
				"Kind": {
					"type": "keyword"
				},
				"meta": {
					"type": "nested",
					"properties": {
						"Name": {
							"type": "text",
							"copy_to": "catch_all"
						},
						"Tenant": {
							"type": "keyword",
							"copy_to": "catch_all"
						}
					},
					"Labels": {
						"type": "nested",
						"properties": {
							"key": {
								"type": "text",
								"copy_to": "catch_all"   <<<<  this is not taking effect
							}
						}
					}
				}
			}
		}
	}
}

I also tried creating a inner meta_catch_all and propagate it to outer catch_call and it didnt help.

Might be a bug. @jpountz WDYT ?