# Did ES / Lucene start tokenizing fields differently in 0.17.0?

**URL:** <https://discuss.elastic.co/t/did-es-lucene-start-tokenizing-fields-differently-in-0-17-0/4384>\
**Category:** Elasticsearch\
**Created:** [May 11, 2011, 5:03pm UTC](https://discuss.elastic.co/t/did-es-lucene-start-tokenizing-fields-differently-in-0-17-0/4384 "2011-05-11T17:03:58Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tsuna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tsuna/32/2897_2.png) [@tsuna](https://discuss.elastic.co/u/tsuna)\
**Post date:** [May 11, 2011, 5:03pm UTC](https://discuss.elastic.co/t/did-es-lucene-start-tokenizing-fields-differently-in-0-17-0/4384/1 "2011-05-11T17:03:58Z")

</div>

Hi all,  
I'm testing my app on elasticsearch-0.17.0-SNAPSHOT (built from  
563ad625c0f69f3ff0f4c39f46421b1dc2c91b6f) and in my app I'm doing a  
term facet on a field. I noticed a difference in behavior. If the  
field contains "foo\_bar", in 0.16 it would be tokenized as 2 tokens  
["foo", "bar"], but in 0.17 it remains a single token ["foo\_bar"]. I  
have absolutely zero configuration change on my ES instance, it's a  
complete vanilla install from the commit above. My mapping is created  
dynamically without me specifying anything about it.

Hence my question: Did ES / Lucene start tokenizing fields differently?

--  
Benoit "tsuna" Sigoure  
Software Engineer @ [www.StumbleUpon.com](http://www.StumbleUpon.com)

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [May 11, 2011, 5:11pm UTC](https://discuss.elastic.co/t/did-es-lucene-start-tokenizing-fields-differently-in-0-17-0/4384/2 "2011-05-11T17:11:08Z")

</div>

Are you sure that in 0.16 it gets tokenized into 2 tokens? I ran the following on 0.15.2, 0.16.0 (where some analysis behavior changed when upgrading to Lucene 0.16.0), and master, and in all of them, it tokenizes into a single token (using the default, standard analyzer).

curl -XPUT localhost:9200/test  
curl -XPOST localhost:9200/test/\_analyze -d 'foo\_bar'

On Wednesday, May 11, 2011 at 8:03 PM, tsuna wrote:

> Hi all,  
> I'm testing my app on elasticsearch-0.17.0-SNAPSHOT (built from  
> 563ad625c0f69f3ff0f4c39f46421b1dc2c91b6f) and in my app I'm doing a  
> term facet on a field. I noticed a difference in behavior. If the  
> field contains "foo\_bar", in 0.16 it would be tokenized as 2 tokens  
> ["foo", "bar"], but in 0.17 it remains a single token ["foo\_bar"]. I  
> have absolutely zero configuration change on my ES instance, it's a  
> complete vanilla install from the commit above. My mapping is created  
> dynamically without me specifying anything about it.
> 
> Hence my question: Did ES / Lucene start tokenizing fields differently?
> 
> --  
> Benoit "tsuna" Sigoure  
> Software Engineer @ [www.StumbleUpon.com](http://www.StumbleUpon.com)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:06am UTC](https://discuss.elastic.co/t/did-es-lucene-start-tokenizing-fields-differently-in-0-17-0/4384/3 "2017-07-06T04:06:28Z")

</div>

Closed on July 6, 2017, 4:06am UTC