# Standard TokenizerFactory deprecated in Elasticsearch Java API v7?

**URL:** <https://discuss.elastic.co/t/standard-tokenizerfactory-deprecated-in-elasticsearch-java-api-v7/223427>\
**Category:** Elasticsearch\
**Created:** [March 12, 2020, 11:45pm UTC](https://discuss.elastic.co/t/standard-tokenizerfactory-deprecated-in-elasticsearch-java-api-v7/223427 "2020-03-12T23:45:34Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![munasia](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@munasia](https://discuss.elastic.co/u/munasia)\
**Post date:** [March 12, 2020, 11:45pm UTC](https://discuss.elastic.co/t/standard-tokenizerfactory-deprecated-in-elasticsearch-java-api-v7/223427/1 "2020-03-12T23:45:34Z")

</div>

Currently performing an Elasticsearch upgrade from v1.7 to v7.1. While refactoring our Java code, I came across this Elasticsearch v1.7 code:

```
    private static final TokenizerFactory STANDARD_TOKENIZER_FACTORY = 
        PreBuiltTokenizers.STANDARD.getTokenizerFactory(Version.V_1_7_3);

    org.apache.lucene.analysis.Tokenizer tokenizer = 
            STANDARD_TOKENIZER_FACTORY.create(new StringReader(terms.toLowerCase()));

```

Can this be recreated using the Java Elasticsearch 7.1 APIs? I've tried to no avail. Looks like`PreBuiltTokenizers.STANDARD.getTokenizerFactory()` has been deprecated in v7.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 13, 2020, 4:43pm UTC](https://discuss.elastic.co/t/standard-tokenizerfactory-deprecated-in-elasticsearch-java-api-v7/223427/2 "2020-03-13T16:43:48Z")

</div>

Hey,

can you maybe explain what you are trying to do and why you need access to those classes? Are you writing a plugin like an `AnalysisPlugin`?

--Alex

---

<div class="post-metadata">

**Author:** ![munasia](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@munasia](https://discuss.elastic.co/u/munasia)\
**Post date:** [March 13, 2020, 8:28pm UTC](https://discuss.elastic.co/t/standard-tokenizerfactory-deprecated-in-elasticsearch-java-api-v7/223427/3 "2020-03-13T20:28:53Z")

</div>

In our Java code, we tokenize search terms then we categorize the tokens into 2 lists: high frequency tokens (like stopwords) and low frequency tokens. Then, we build a compound query and send it to Elasticsearch. In the query, the 2 frequencies have different settings applied to them (e.g different boosts etc). This is legacy code.

For example if the user searches for "the quick and fast brown fox", we'll break that up into ["the and"] and ["quick fast brown"], then create a query (that boosts low frequency tokens different than high frequency tokens) and send it to Elasticsearch.

This is just one example. That tokenizing logic that I have in my original post is in a common util method that is used in multiple places for the purpose of tokenizing strings.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 16, 2020, 1:50pm UTC](https://discuss.elastic.co/t/standard-tokenizerfactory-deprecated-in-elasticsearch-java-api-v7/223427/4 "2020-03-16T13:50:50Z")

</div>

You may want to read about the [common terms query](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/query-dsl-common-terms-query.html) - but even more important, why it is deprecated in newer versions, that might help you to rethink your whole architecture in that regard.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 13, 2020, 1:50pm UTC](https://discuss.elastic.co/t/standard-tokenizerfactory-deprecated-in-elasticsearch-java-api-v7/223427/5 "2020-04-13T13:50:52Z")

</div>

Closed on April 13, 2020, 1:50pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.