# Using a Python Tokenizer

**URL:** <https://discuss.elastic.co/t/using-a-python-tokenizer/117656>\
**Category:** Elasticsearch\
**Created:** [January 30, 2018, 4:02pm UTC](https://discuss.elastic.co/t/using-a-python-tokenizer/117656 "2018-01-30T16:02:22Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![chbrown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chbrown/32/99368_2.png) [@chbrown](https://discuss.elastic.co/u/chbrown)\
**Post date:** [January 30, 2018, 4:02pm UTC](https://discuss.elastic.co/t/using-a-python-tokenizer/117656/1 "2018-01-30T16:02:23Z")

</div>

Hi there!

Wondering if anyone has tried using a custom tokenizer written in python?

What are some possible approaches? I'm guessing it would need to be added via a custom plugin?

We currently have one offered to us for segmenting a few Asian languages, and are wondering if its possible to use with elasticsearch.

Chris

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [January 30, 2018, 4:23pm UTC](https://discuss.elastic.co/t/using-a-python-tokenizer/117656/2 "2018-01-30T16:23:49Z")

</div>

Custom analysis code/plugins are run within the same JVM used by  
Elasticsearch (jwith a separate classloader), so the code must be written  
in a JVM language. This limitation is due to the fact that the code is  
required by the Lucene library, which is written in Java. I have written  
plugins in Java and Scala. Python only works as a scripting language.  
Someone please correct me if I am wrong, there always seems to be a  
workaround.

---

<div class="post-metadata">

**Author:** ![chbrown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chbrown/32/99368_2.png) [@chbrown](https://discuss.elastic.co/u/chbrown)\
**Post date:** [January 30, 2018, 4:32pm UTC](https://discuss.elastic.co/t/using-a-python-tokenizer/117656/3 "2018-01-30T16:32:41Z")

</div>

Thanks Ivan! Everything I've found online is telling me the same.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 27, 2018, 4:32pm UTC](https://discuss.elastic.co/t/using-a-python-tokenizer/117656/4 "2018-02-27T16:32:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.