# Elasticsearch: Testing Analyzers

**URL:** <https://discuss.elastic.co/t/elasticsearch-testing-analyzers/151757>\
**Category:** Elasticsearch\
**Created:** [October 10, 2018, 6:27am UTC](https://discuss.elastic.co/t/elasticsearch-testing-analyzers/151757 "2018-10-10T06:27:46Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [October 10, 2018, 6:27am UTC](https://discuss.elastic.co/t/elasticsearch-testing-analyzers/151757/1 "2018-10-10T06:27:46Z")

</div>

Hi Team

Can someone please give insights on how to test the Analyzers?

Once we define Analyzers in mappings and started indexing, how should we test the Analyzers? We want to understand the way how each analyzer performs and which analyzer works best for our data.

How to understand this? Please suggest

Thanks for your time as always:🙂

-rahul

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 10, 2018, 6:57am UTC](https://discuss.elastic.co/t/elasticsearch-testing-analyzers/151757/2 "2018-10-10T06:57:02Z")

</div>

Use the \_analyze API

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [October 10, 2018, 7:49am UTC](https://discuss.elastic.co/t/elasticsearch-testing-analyzers/151757/3 "2018-10-10T07:49:24Z")

</div>

Will do that. @dadoonet

Thank you.

In addition, any insights on difference between adding analyzer in settings and mappings?

You can see it here

In Settings:

```
PUT my_index
{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_english_analyzer": {
          "type": "standard",
          "max_token_length": 5,
          "stopwords": "_english_"
        }
      }
    }
  }
} 

```

In Mappings:

```
PUT my_index
{
  "mappings": {
    "_doc": {
      "properties": {
        "title": {
          "type": "text",
          "analyzer": "standard"
        }
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [October 12, 2018, 4:47am UTC](https://discuss.elastic.co/t/elasticsearch-testing-analyzers/151757/4 "2018-10-12T04:47:37Z")

</div>

Hi @dadoonet

As per my understanding settings is for additional configuration of analyzers.

Also, If we need to apply tokenizers we should do that along with analyzers in the settings. Am I right?

And In settings I can see Index\_Analyzer and Search\_Analyzer. Can you please differentiate both. As per the name I can tell one is analyzed at index time and the other at search time. Is search\_analyzer to analyze the query ?

Requirement: Should be apply to Analyzers, tokenizers to improve the relevancy.

Thanks

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 12, 2018, 9:22am UTC](https://discuss.elastic.co/t/elasticsearch-testing-analyzers/151757/5 "2018-10-12T09:22:14Z")

</div>

If you want to define a custom analyzer, do that in settings.  
If you want to apply an analyzer to a field, you do that in mapping.

By default the analyzer is used at index time and search time. But you can optionally specify another analyzer to apply at search time.

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [October 12, 2018, 9:43am UTC](https://discuss.elastic.co/t/elasticsearch-testing-analyzers/151757/6 "2018-10-12T09:43:13Z")

</div>

got it @dadoonet

Thanks much for your time.

So, I think it's not necessary for us to process/enrich the user query as we are already applying analysers/tokenizer on target fields.

For instance, consider this search term(user query): **I have an issue with the internet, how to solve?**

do we need to remove stop words even in user query to increase the relevancy? which will convert the above search term to **I have issue with internet**

does elasticsearch have any features to support query processing?(optimizing the user query to increase the relevancy)

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 12, 2018, 10:24am UTC](https://discuss.elastic.co/t/elasticsearch-testing-analyzers/151757/7 "2018-10-12T10:24:24Z")

</div>

The default settings in elasticsearch are pretty much good. BM25 algorithm takes care about stop words OOTB.

Just give it a try and you'll see how good it is and if it needs some tuning or not.

---

<div class="post-metadata">

**Author:** ![rahulnama](https://avatars.discourse-cdn.com/v4/letter/r/8c91f0/32.png) [@rahulnama](https://discuss.elastic.co/u/rahulnama)\
**Post date:** [October 15, 2018, 4:58am UTC](https://discuss.elastic.co/t/elasticsearch-testing-analyzers/151757/8 "2018-10-15T04:58:34Z")

</div>

Sure @dadoonet

Read about BM25 in documentation but never tried it. Will give it a try

Thank you much 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 12, 2018, 4:58am UTC](https://discuss.elastic.co/t/elasticsearch-testing-analyzers/151757/9 "2018-11-12T04:58:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.