# Elasticsearch: search and index time analyzer

**URL:** <https://discuss.elastic.co/t/elasticsearch-search-and-index-time-analyzer/165028>\
**Category:** Elasticsearch\
**Created:** [January 21, 2019, 10:26am UTC](https://discuss.elastic.co/t/elasticsearch-search-and-index-time-analyzer/165028 "2019-01-21T10:26:49Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [January 21, 2019, 10:26am UTC](https://discuss.elastic.co/t/elasticsearch-search-and-index-time-analyzer/165028/1 "2019-01-21T10:26:49Z")

</div>

Hi Team

I'm using Custom Analyzer. But, I want to use the analyzer at index time and search time as well. I've mentioned in mappings but I dont see that search time analyzer is working?

Below are settings. I'm using Analyzer for content field. (please refer to content field in mappings)

> ```
> "settings": {
> "number_of_shards" : 1,
> "analysis": {
> "analyzer": {
> "my_analyzer": {
> "type": "custom",
> "tokenizer": "standard",
> "filter": [
> "possessive_stemmer",
> "lowercase",
> "english_stop",
> "eng_keywords",
> "stemmer"
> ]
> }
> },
> "filter": {
> "english_stop": {
> "type": "stop",
> "stopwords": ["have","should","i","a", "an", "and", "are", "as", "at", "be", "but", "by", "for", "if", "in", "into", "is", "it", "no", "not", "of", "on", "or", "such", "that", "the", "their", "then", "there", "these", "they", "this", "to", "was", "will", "with","my"]
> },
> "stemmer": {
> "type": "stemmer",
> "language": "light_english"
> },
> "possessive_stemmer": {
> "type": "stemmer",
> "language": "possessive_english"
> },
> "eng_keywords": {
> "type": "keyword_marker",
> "keywords": [
> "windows"
> ]
> }
> }
> }
> },
> "mappings": {
> "_doc": {
> "properties": {
> "Author": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "CreationDate": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "Creator": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "Keywords": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "ModDate": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "Producer": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "Subject": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "Title": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "content": {
> "type": "text",
> "analyzer": "my_analyzer",
> "search_analyzer": "my_analyzer",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "file_category": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "file_name": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> },
> "url": {
> "type": "text",
> "fields": {
> "keyword": {
> "type": "keyword",
> "ignore_above": 256
> }
> }
> }
> }
> }
> }
> 
> ```

When I search for a query. Example: **my monitor is not running.**

According to explain api, ES is searching for running instead of run(as I'm using stemmer).

Please let me know what I'm missing here ?

Thanks 🙂

---

<div class="post-metadata">

**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\
**Post date:** [January 22, 2019, 1:21pm UTC](https://discuss.elastic.co/t/elasticsearch-search-and-index-time-analyzer/165028/3 "2019-01-22T13:21:54Z")

</div>

What query are you using? Maybe you can share the exact query here?

If you're using a `term` query, then no analysis will be applied to your search terms, even if you have specified a search\_analyzer (this is the behavior of the term query). In that case, you would need to switch to the `match` query instead.

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [January 23, 2019, 1:04pm UTC](https://discuss.elastic.co/t/elasticsearch-search-and-index-time-analyzer/165028/4 "2019-01-23T13:04:40Z")

</div>

hi @abdon. Thanks for connecting

I'm using match query. here is the query

```
> { "_source": "url", 
> "explain": true, 
> "query": {
> "match" : {
> "content" : {
> "query" : "my keyboard is not running"
> }
> }
> }
> } 

```

**response:**

```
> "hits": {
> "total": 17,
> "max_score": 2.021533,
> "hits": [
> {
> "_shard": "[newoneindex][0]",
> "_node": "nTOGuiS3SsGXFeD5Bf3FxQ",
> "_index": "newoneindex",
> "_type": "_doc",
> "_id": "6",
> "_score": 2.021533,
> "_source": {
> "url": "/Linux/linux_faq_4_manual.pdf"
> },
> "_explanation": {
> "value": 2.0215333,
> "description": "sum of:",
> "details": [
> {
> "value": 1.0470022,
> "description": "weight(content:keyboard in 2) [PerFieldSimilarity], result of:",
> "details": [
> {
> "value": 1.0470022,
> "description": "score(doc=2,freq=3.0 = termFreq=3.0\n), product of:",
> "details": [
> {
> "value": 0.6931472,
> "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
> "details": [
> {
> "value": 10,
> "description": "docFreq",
> "details": []
> },
> {
> "value": 20,
> "description": "docCount",
> "details": []
> }
> ]
> },
> {
> "value": 1.5105048,
> "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
> "details": [
> {
> "value": 3,
> "description": "termFreq=3.0",
> "details": []
> },
> {
> "value": 1.2,
> "description": "parameter k1",
> "details": []
> },
> {
> "value": 0.75,
> "description": "parameter b",
> "details": []
> },
> {
> "value": 4760.05,
> "description": "avgFieldLength",
> "details": []
> },
> {
> "value": 5656,
> "description": "fieldLength",
> "details": []
> }
> ]
> }
> ]
> }
> ]
> },
> {
> "value": 0.974531,
> "description": "weight(content:running in 2) [PerFieldSimilarity], result of:",
> "details": [
> {
> "value": 0.974531,
> "description": "score(doc=2,freq=8.0 = termFreq=8.0\n), product of:",
> "details": [
> {
> "value": 0.5187938,
> "description": "idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from:",
> "details": [
> {
> "value": 12,
> "description": "docFreq",
> "details": []
> },
> {
> "value": 20,
> "description": "docCount",
> "details": []
> }
> ]
> },
> {
> "value": 1.8784553,
> "description": "tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from:",
> "details": [
> {
> "value": 8,
> "description": "termFreq=8.0",
> "details": []
> },
> {
> "value": 1.2,
> "description": "parameter k1",
> "details": []
> },
> {
> "value": 0.75,
> "description": "parameter b",
> "details": []
> },
> {
> "value": 4760.05,
> "description": "avgFieldLength",
> "details": []
> },
> {
> "value": 5656,
> "description": "fieldLength",
> "details": []
> }
> ]
> }
> ]
> }
> ]
> }
> ]
> }
> }
```

---

<div class="post-metadata">

**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\
**Post date:** [January 23, 2019, 4:40pm UTC](https://discuss.elastic.co/t/elasticsearch-search-and-index-time-analyzer/165028/5 "2019-01-23T16:40:49Z")

</div>

The `light_english` stemmer that you're using does not actually stem `running` to `run`. You can see that by using the `_analyze` API:

```auto
GET newoneindex/_analyze
{
  "analyzer": "my_analyzer",
  "text": "my keyboard is not running"
}

```

If you replace the `light_english` stemmer by for example the `english` stemmer, you will see that `running` is actually stemmed to `run`.

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [January 29, 2019, 6:09am UTC](https://discuss.elastic.co/t/elasticsearch-search-and-index-time-analyzer/165028/6 "2019-01-29T06:09:49Z")

</div>

@abdon

Yea I missed that. Thank you : )

In addition, any analyzer for handling keywords like haven't, shouldn't, can't etc. All this should be transformed to have not, should not, can not etc.  
and should be removed if they are in stopwords list.

-Rahul

---

<div class="post-metadata">

**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\
**Post date:** [January 29, 2019, 12:25pm UTC](https://discuss.elastic.co/t/elasticsearch-search-and-index-time-analyzer/165028/7 "2019-01-29T12:25:20Z")

</div>

I don't know of an easy way to do that. Maybe a [mapping character filter](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-mapping-charfilter.html) could be the way to go?

Maybe you can open a new topic on this forum to give your question some visibility?

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [January 29, 2019, 12:37pm UTC](https://discuss.elastic.co/t/elasticsearch-search-and-index-time-analyzer/165028/8 "2019-01-29T12:37:33Z")

</div>

I should check mapping character filter

Sure @abdon

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 26, 2019, 12:37pm UTC](https://discuss.elastic.co/t/elasticsearch-search-and-index-time-analyzer/165028/9 "2019-02-26T12:37:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.