# Wildcard queries and custom templates

**URL:** <https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820>\
**Category:** Elasticsearch\
**Created:** [November 11, 2016, 2:28pm UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820 "2016-11-11T14:28:12Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![trondhindenes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trondhindenes/32/10534_2.png) [@trondhindenes](https://discuss.elastic.co/u/trondhindenes)\
**Post date:** [November 11, 2016, 2:28pm UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/1 "2016-11-11T14:28:12Z")

</div>

hi,  
We're in the process of moving from the logstash builtin template to our own in order to have more control of our fields and data types. I'm seeing some problems with wildcard searches using our new index that I'd very much like to some expert advise on.  
Firstly, our es template looks like this:

```javascript
{
  "company_app_log": {
    "order": 0,
    "template": "company_app_log*",
    "settings": {
      "index": {
        "refresh_interval": "5s"
      }
    },
    "mappings": {
      "formattederror": {
        "dynamic_templates": [
          {
            "notanalyzed": {
              "mapping": {
                "index": "not_analyzed",
                "type": "string"
              },
              "match_mapping_type": "string",
              "match": "*"
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "errorid": {
            "type": "integer"
          }
        }
      },
      "telemetry": {
        "dynamic_templates": [
          {
            "notanalyzed": {
              "mapping": {
                "index": "not_analyzed",
                "type": "string"
              },
              "match_mapping_type": "string",
              "match": "*"
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          }
        }
      },
      "http_measurement": {
        "dynamic_templates": [
          {
            "notanalyzed": {
              "mapping": {
                "index": "not_analyzed",
                "type": "string"
              },
              "match_mapping_type": "string",
              "match": "*"
            }
          }
        ],
        "properties": {
          "httpstatuscode": {
            "type": "integer"
          },
          "@timestamp": {
            "type": "date"
          },
          "elapsedmilliseconds": {
            "type": "integer"
          }
        }
      }
    },
    "aliases": {}
  }
}

```

One document in the index looks like this:

```javascript
{
	"_index": "company_log-uat-2016.11",
	"_type": "http_measurement",
	"_id": "AVhTY6gacmfALXNtkch5",
	"_score": null,
	"_source": {
		"Level": "Information",
		"@version": "1",
		"@timestamp": "2016-11-11T11:42:48.481Z",
		"count": 1,
		"fields": null,
		"beat": {
			"hostname": "Z49OS2SWB114T",
			"name": "Z49OS2SWB114T"
		},
		"input_type": "log",
		"tags": [
			"environment:uat",
			"servicefamily:services-external",
			"servicegroup:api",
			"beats_input_codec_json_applied",
			"http_measurement"
		],
		"offset": 3615,
		"type": "log",
		"host": "Z49OS2SWB114T",
		"logtype": "http_measurement",
		"computername": "Z49OS2SWB114T",
		"httpmethod": "GET",
		"requesturi": "https://company.com/api/web/asset/1595262/play?protocol=HLS",
		"httpstatusstring": "OK",
		"httpstatuscode": 200,
		"elapsedmilliseconds": 221
	}
}

```

I'm struggling to understand how I can wildcard search in Kibana using this data.  
In regular "logstashed" indices I could for eample use this query:

```auto
requesturi.raw: "*web*"

```

but the corresponding

```auto
requesturi: "*web*"

```

doesnt provide any results from this index. If I remove the quotation marks it works, but that leaves me unable to search using any terms containing special characters, such as:

```auto
requesturi: "*api/web/asset/*"

```

Am I doing something wrong in my template?  
For the record, we're not on ELK 5.0, we're still running the previous version of the stack.

---

<div class="post-metadata">

**Author:** ![trondhindenes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trondhindenes/32/10534_2.png) [@trondhindenes](https://discuss.elastic.co/u/trondhindenes)\
**Post date:** [November 11, 2016, 5:42pm UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/2 "2016-11-11T17:42:05Z")

</div>

I dug a bit deeper into this, and it turns out queries on an index where the default logstash template it used will have an analysed field "thing" and a non-analyzed field representing the same data ("thing.raw"). The non-analysed field shows the same behaviors as fields where my custom template is used, which is to be expected since they're both un-analyzed string fields.

What I struggle to understand is: I seem to be able to perform wildcard queries on a non-analyzed field, as long as I don't use any special characters. So, my question is: Do I have to use an analyzed field just to have support for wildcard queries containing special characters (like "/")?

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [November 11, 2016, 5:54pm UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/3 "2016-11-11T17:54:53Z")

</div>

Are you sure your template is resulting in an un-analyzed string field for `requesturi`? You can ask ES for the mappings on the index to confirm what the template actually did.

I ask that because the default analyzer would split on characters like / which might explain why wildcards with / are not working.

You must use an un-analyzed field if you want wildcards like `*api/web/asset*` to work.

Mike McCandless

---

<div class="post-metadata">

**Author:** ![trondhindenes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trondhindenes/32/10534_2.png) [@trondhindenes](https://discuss.elastic.co/u/trondhindenes)\
**Post date:** [November 11, 2016, 6:24pm UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/4 "2016-11-11T18:24:52Z")

</div>

Thanks for responding Mike,  
I can't seen to find anything in the documentation regarding how to list templates for a given index. I can list them globally, that's all.

However, like I wrote above a "regular" index using Logstash's built-in template shows the same symptom on the "raw" field (which is also non-analyzed) - I'm unable to use special characters together with wildcard searches on a "raw" field.

For example, this doesn't work (this is an index using the defualt logstash template):

```auto
uriStem.raw: "/client/1/*"

```

However, on the same index, this works fine:

```auto
uriStem.raw: "/client/1/catchup"

```

On the analyzed "version" of the same field, this works:

```auto
uriStem: "/client/1/*"

```

This seems to be the opposite of your explanation? (Not saying it's wrong, just trying to understand the behavior)

---

<div class="post-metadata">

**Author:** ![trondhindenes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trondhindenes/32/10534_2.png) [@trondhindenes](https://discuss.elastic.co/u/trondhindenes)\
**Post date:** [November 11, 2016, 6:38pm UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/5 "2016-11-11T18:38:29Z")

</div>

So I managed to find the mappings for the index, and everything looks correct there:  
(subset):

```auto
          "messagelevel": {
            "type": "string",
            "index": "not_analyzed"
          },
          "offset": {
            "type": "long"
          },
          "requesturi": {
            "type": "string",
            "index": "not_analyzed"
          },
          "source": {
            "type": "string",
            "index": "not_analyzed"
          },
          "sourcecontext": {
            "type": "string",
            "index": "not_analyzed"
          },
          "tags": {
            "type": "string",
            "index": "not_analyzed"
          },
          "type": {
            "type": "string",
            "index": "not_analyzed"

```

---

<div class="post-metadata">

**Author:** ![trondhindenes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trondhindenes/32/10534_2.png) [@trondhindenes](https://discuss.elastic.co/u/trondhindenes)\
**Post date:** [November 11, 2016, 6:46pm UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/6 "2016-11-11T18:46:51Z")

</div>

...and Kibana agrees aswell:

 ![](https://us1.discourse-cdn.com/elastic/original/2X/0/0bafc6158149bae3e080f32d5e4cac93e07a2ae1.png)

---

<div class="post-metadata">

**Author:** ![trondhindenes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trondhindenes/32/10534_2.png) [@trondhindenes](https://discuss.elastic.co/u/trondhindenes)\
**Post date:** [November 11, 2016, 6:53pm UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/7 "2016-11-11T18:53:50Z")

</div>

So, I guess it boils down to: How come this works:

```javascript
          "query": "sourcecontext: Api.Play.Agents.PlayAgent"

```

but this doesnt (eg, it gives zero hits?

```javascript
          "query": "sourcecontext: Api.Play.Agents.Play*"

```

The field is confirmed string and "not analyzed"

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [November 11, 2016, 7:03pm UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/8 "2016-11-11T19:03:03Z")

</div>

Hmm, instead of passing a query string at search time, can you use the DSL queries to explicitly construct a wildcard query? This just simplifies the problem to bypass any analysis the query parser may be doing.

And yes the situation is indeed opposite of what I expect 🙂

Mike McCandless

---

<div class="post-metadata">

**Author:** ![trondhindenes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trondhindenes/32/10534_2.png) [@trondhindenes](https://discuss.elastic.co/u/trondhindenes)\
**Post date:** [November 11, 2016, 7:06pm UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/9 "2016-11-11T19:06:09Z")

</div>

_I_ can, but I can't expect my users do do use DSL queries instead of Kibana. It seems that (part of) the problem is how Kibana constructs it's queries. But you're right, the wildcard query type using the query api works perfectly.

Still not sure how to solve this one.

---

<div class="post-metadata">

**Author:** ![trondhindenes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trondhindenes/32/10534_2.png) [@trondhindenes](https://discuss.elastic.co/u/trondhindenes)\
**Post date:** [November 11, 2016, 7:10pm UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/10 "2016-11-11T19:10:13Z")

</div>

Just for ref, this seems to force Kibana to issue a wildcard query:

```javascript
{"wildcard":{"sourcecontext": "Stuff*"}

```

However, leading wildcards do not seem to work:

```javascript
{"wildcard":{"sourcecontext": "*uff*"}

```

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [November 11, 2016, 11:30pm UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/11 "2016-11-11T23:30:43Z")

</div>

OK the fact that the DSL query produces the correct results is good news ... it means something is going wrong when parsing your query.

Does anyone know if ES gives you a way to see what exact (Lucene level) query was created by parsing the query string?

Anyway, by default, wildcard terms are not analyzed by the query parser behind query string queries, so it's odd it's not working with query string.

But `/` is a [reserved character](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_reserved_characters) ... try escaping it by prefixing with `\`?

Separately, I'm not sure why you see leading wildcard NOT working w/ a DSL query. Does it also not work if you run the query directly against ES (not through Kibana)?

Note that leading wildcard queries are exceptionally costly to run ... it requires Lucene to scan every unique term in the index.

Mike McCandless

---

<div class="post-metadata">

**Author:** ![trondhindenes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trondhindenes/32/10534_2.png) [@trondhindenes](https://discuss.elastic.co/u/trondhindenes)\
**Post date:** [November 14, 2016, 8:27am UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/12 "2016-11-14T08:27:31Z")

</div>

Just to keep updating this thread in case anyone else stumbles upon the same: The following query seems to work in Kibana, and caters for special characters:

```javascript
 "query": "requesturi: *\"web/asset*\"

```

That is, keep the wildcard outside the quotation marks which represents your query  
In Kibana, that would be typed as:

```auto
requesturi: *"web/asset*"

```

This also seems to work:

```auto
requesturi: *"web/asset"*

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 12, 2016, 8:27am UTC](https://discuss.elastic.co/t/wildcard-queries-and-custom-templates/65820/13 "2016-12-12T08:27:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.