# Custom Elasticsearch Analyzer :- Tokenization and Detokenize Text Processing

**URL:** <https://discuss.elastic.co/t/custom-elasticsearch-analyzer-tokenization-and-detokenize-text-processing/360314>\
**Category:** Elasticsearch\
**Created:** [May 27, 2024, 4:50pm UTC](https://discuss.elastic.co/t/custom-elasticsearch-analyzer-tokenization-and-detokenize-text-processing/360314 "2024-05-27T16:50:01Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![alliswell](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alliswell/32/117514_2.png) [@alliswell](https://discuss.elastic.co/u/alliswell)\
**Post date:** [May 27, 2024, 4:50pm UTC](https://discuss.elastic.co/t/custom-elasticsearch-analyzer-tokenization-and-detokenize-text-processing/360314/1 "2024-05-27T16:50:01Z")

</div>

I am working on building a custom analyzer that needs to implement a unique text processing workflow. Here’s a breakdown of the required steps:

1. **Tokenize** the input text based on whitespace.

2. **Stem** each token using an English stemmer.

3. **Detokenize** the tokens by merging them back into a single string.

4. **Re-tokenize** the resulting string based on the presence of hash ('#') characters.

**Example Process:**

- **Input Text:**"athletic shoes # running shoes"

- **After Step 1 (Tokenize by whitespace):** ["athletic", "shoes", "#", "running", "shoes"]

- **After Step 2 (Stemming):** ["athlet", "shoe", "#", "run", "shoe"]

- **After Step 3 (Detokenize):**"athlet shoe # run shoe"

- **After Step 4 (Tokenize by '#'):** ["athlet shoe ", "run shoe"]

Is it possible to achieve step 3 (detokenize) using the existing Elasticsearch inbuilt functionalities?

Since what I'm trying to do is non-liner text processing, I'm not sure if it's possible to achieve this using the existing Elasticsearch inbuilt functionalities. Just asking here before I start developing a custom plugin or pre-processing the text outside of Elasticsearch.