# Jumped from 1.X to 2.X and noticed my tokenization changed

**URL:** <https://discuss.elastic.co/t/jumped-from-1-x-to-2-x-and-noticed-my-tokenization-changed/55244>\
**Category:** Elasticsearch\
**Created:** [July 11, 2016, 11:26pm UTC](https://discuss.elastic.co/t/jumped-from-1-x-to-2-x-and-noticed-my-tokenization-changed/55244 "2016-07-11T23:26:58Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shadoninja](https://avatars.discourse-cdn.com/v4/letter/s/71e660/32.png) [@Shadoninja](https://discuss.elastic.co/u/Shadoninja)\
**Post date:** [July 11, 2016, 11:26pm UTC](https://discuss.elastic.co/t/jumped-from-1-x-to-2-x-and-noticed-my-tokenization-changed/55244/1 "2016-07-11T23:26:58Z")

</div>

Here is an old entry:

```
}, {
  "_index" : "logs-2014-12-11",
  "_type" : "logs",
  "_id" : "505",
  "_score" : 1.0,
  "fields" : {
    "terms" : [[ "smith/robert/principal/janitor"] ]
  }
}, {

```

Here is the new entry:

```
}, {
  "_index" : "logs-2014-12-11",
  "_type" : "logs",
  "_id" : "505",
  "_score" : 1.0,
  "fields" : {
    "terms" : ["janitor", "principal", "robert", "smith"]
  }
}, {

```

Is this something that is supposed to happen? I know a huge amount of the library changed on the update.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 12, 2016, 4:25am UTC](https://discuss.elastic.co/t/jumped-from-1-x-to-2-x-and-noticed-my-tokenization-changed/55244/2 "2016-07-12T04:25:59Z")

</div>

What is the mapping in both cases?

---

<div class="post-metadata">

**Author:** ![Shadoninja](https://avatars.discourse-cdn.com/v4/letter/s/71e660/32.png) [@Shadoninja](https://discuss.elastic.co/u/Shadoninja)\
**Post date:** [July 12, 2016, 6:08pm UTC](https://discuss.elastic.co/t/jumped-from-1-x-to-2-x-and-noticed-my-tokenization-changed/55244/3 "2016-07-12T18:08:16Z")

</div>

Figuring that out now.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 12, 2016, 8:26pm UTC](https://discuss.elastic.co/t/jumped-from-1-x-to-2-x-and-noticed-my-tokenization-changed/55244/4 "2016-07-12T20:26:04Z")

</div>

Would be better to give to others the solution in case someone else hit this.

---

<div class="post-metadata">

**Author:** ![Shadoninja](https://avatars.discourse-cdn.com/v4/letter/s/71e660/32.png) [@Shadoninja](https://discuss.elastic.co/u/Shadoninja)\
**Post date:** [July 12, 2016, 9:53pm UTC](https://discuss.elastic.co/t/jumped-from-1-x-to-2-x-and-noticed-my-tokenization-changed/55244/5 "2016-07-12T21:53:26Z")

</div>

The mapping used on it looks to be this:

```
"login": {
   "type": "string",
   "store": true
},

```

The raw message when queried directly from elasticsearch looks like this on BOTH ES 1.7 and 2.3.3:

```
"_index" : "logs-2014-12-11",
"_type" : "logs",
"_id" : "505",
"_score" : 1.0,
"_source" : {
  "login" : "smith/robert/principal/janitor",
  "msgClassId" : 3510,
  "logMessage" : "AllPatternMatchesForWord.json",
  "normalDate" : "2014-11-01T18:38:35.0305041Z"

```

}

The original post that showed differences in the entires was generated using a groovy script trick that was supposed to show the tokenization done by es (I will admit I don't understand it much):

tokens.txt:

```
{
   "query" : {
        "match_all" : {}
    },
    "script_fields":{
          "terms" : {
              "script": "doc[field].values",
              "params": {
                  "field": "login"
               }
       }
    }

```

}

```
curl localhost:9200/logs-2014-12-11/_search?pretty -d @C:\temp\tokens.txt

```

Settings changed to make it work:

es 1.7:

```
script.groovy.sandbox.enabled: true (1.7)

```

es 2.3:

```
script.engine.groovy.inline.search: on (2.3)
```

---

<div class="post-metadata">

**Author:** ![Shadoninja](https://avatars.discourse-cdn.com/v4/letter/s/71e660/32.png) [@Shadoninja](https://discuss.elastic.co/u/Shadoninja)\
**Post date:** [July 29, 2016, 9:12pm UTC](https://discuss.elastic.co/t/jumped-from-1-x-to-2-x-and-noticed-my-tokenization-changed/55244/6 "2016-07-29T21:12:52Z")

</div>

Ah I finally figured it out! ES 2.x doesn't automatically load the template files! Once I put them in manually, everything started working again!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:31pm UTC](https://discuss.elastic.co/t/jumped-from-1-x-to-2-x-and-noticed-my-tokenization-changed/55244/7 "2017-07-05T22:31:37Z")

</div>