# Configuring the standard tokenizer elasticsearch

**URL:** <https://discuss.elastic.co/t/configuring-the-standard-tokenizer-elasticsearch/150742>\
**Category:** Elasticsearch\
**Created:** [October 2, 2018, 5:20pm UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer-elasticsearch/150742 "2018-10-02T17:20:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![surajdalvi](https://avatars.discourse-cdn.com/v4/letter/s/85f322/32.png) [@surajdalvi](https://discuss.elastic.co/u/surajdalvi)\
**Post date:** [October 2, 2018, 5:20pm UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer-elasticsearch/150742/1 "2018-10-02T17:20:52Z")

</div>

Hi,

I am using default tokenizer(standard) for my index in elastic search. and adding documents to it. but standard tokenizer can't split words which having "." dot in it. For example:

```auto
POST _analyze
{
  "tokenizer": "standard",
  "text": "pink.jpg"
}

```

Gives me the response as:

```auto
{
  "tokens": [
    {
      "token": "pink.jpg",
      "start_offset": 0,
      "end_offset": 8,
      "type": "<ALPHANUM>",
      "position": 0
    }
  ]
}

```

The above response showing the whole word in one term. Can we divide it into two terms using "."(dot) operator in standard tokenizer? is any setting in standard tokenizer for this?

---

<div class="post-metadata">

**Author:** ![jaddison](https://avatars.discourse-cdn.com/v4/letter/j/e5b9ba/32.png) [@jaddison](https://discuss.elastic.co/u/jaddison)\
**Post date:** [October 2, 2018, 8:10pm UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer-elasticsearch/150742/2 "2018-10-02T20:10:16Z")

</div>

Use different tokenizers. Look at [https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-letter-tokenizer.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-letter-tokenizer.html) and [https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-chargroup-tokenizer.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-chargroup-tokenizer.html).

The latter might be more appropriate.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 30, 2018, 8:10pm UTC](https://discuss.elastic.co/t/configuring-the-standard-tokenizer-elasticsearch/150742/3 "2018-10-30T20:10:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.