# Ingest Processor Plugin with Analyzer Support

**URL:** <https://discuss.elastic.co/t/ingest-processor-plugin-with-analyzer-support/123365>\
**Category:** Elasticsearch\
**Created:** [March 10, 2018, 12:19am UTC](https://discuss.elastic.co/t/ingest-processor-plugin-with-analyzer-support/123365 "2018-03-10T00:19:53Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![purefield](https://avatars.discourse-cdn.com/v4/letter/p/57b2e6/32.png) [@purefield](https://discuss.elastic.co/u/purefield)\
**Post date:** [March 10, 2018, 12:19am UTC](https://discuss.elastic.co/t/ingest-processor-plugin-with-analyzer-support/123365/1 "2018-03-10T00:19:53Z")

</div>

We are working on building an ingest processor plugin to support the use of analyzer for tokenization. This can be used to create word clouds from text.

How does one access an Analyzer to perform this task of tokenizing a text? Here is what we have so far.  
Utilizing ideas and works from:

- [https://github.com/elastic/elasticsearch/pull/20215/files](https://github.com/elastic/elasticsearch/pull/20215/files)

- [https://github.com/elastic/elasticsearch/pull/20233/files](https://github.com/elastic/elasticsearch/pull/20233/files)

- [https://github.com/spinscale/elasticsearch-ingest-opennlp](https://github.com/spinscale/elasticsearch-ingest-opennlp)

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [March 12, 2018, 4:06pm UTC](https://discuss.elastic.co/t/ingest-processor-plugin-with-analyzer-support/123365/2 "2018-03-12T16:06:02Z")

</div>

I think the code looks good. Is there something that is not working?

If you want to create a word cloud, it is possible that an analyzer could  
emit tokens that are not legible if you use "destructive" filters such as  
stemming and ngrams. On the generic code side, since order does not matter  
in a word cloud, I would return a set or a counting map instead of a list  
(and return a value, not mutate a parameter as a side effect).

---

<div class="post-metadata">

**Author:** ![purefield](https://avatars.discourse-cdn.com/v4/letter/p/57b2e6/32.png) [@purefield](https://discuss.elastic.co/u/purefield)\
**Post date:** [March 12, 2018, 6:53pm UTC](https://discuss.elastic.co/t/ingest-processor-plugin-with-analyzer-support/123365/3 "2018-03-12T18:53:01Z")

</div>

The code works. The concern with the code is the need to access an analyzer. The only way that seems possible is to use the parameters passed in. While this works for the execution it does not work for the test class as we would have to mock the parameters to instantiate TextTokenizerProcessor. How can we test the analyzer properly?

---

<div class="post-metadata">

**Author:** ![purefield](https://avatars.discourse-cdn.com/v4/letter/p/57b2e6/32.png) [@purefield](https://discuss.elastic.co/u/purefield)\
**Post date:** [March 12, 2018, 9:44pm UTC](https://discuss.elastic.co/t/ingest-processor-plugin-with-analyzer-support/123365/4 "2018-03-12T21:44:02Z")

</div>

Creating a new pipeline with the processor using a defined custom analyzer fails:

```bash
curl -sH 'content-type: application/json' -X PUT http://$es_node:9200/_ingest/pipeline/text-tokenizer-pipeline -d '{
> "description": "A pipeline to tokenize text into array",
> "processors": [
> {
> "text_tokenizer": {
> "source_field": "text",
> "target_field": "word_cloud",
> "analyzer": "custom_standard_lowercase_analyzer",
> "on_failure" : [
> {
> "set" : {
> "field" : "error",
> "value" : "field \"text\" does not exist, cannot create \"word_cloud\""
> }
> }
> ]
> }
> },
> {
> "text_tokenizer": {
> "source_field": "post1",
> "target_field": "words1",
> "analyzer": "stop"
> }
> },
> {
> "text_tokenizer": {
> "source_field": "post2",
> "target_field": "words2",
> "analyzer": "simple"
> }
> },
> {
> "text_tokenizer": {
> "source_field": "post3",
> "target_field": "words3",
> "analyzer": "standard"
> }
> }
> ]
> }' | jq
{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "java.lang.IllegalArgumentException: Analyzer [custom_standard_lowercase_analyzer] not found.",
        "header": {
          "processor_type": "text_tokenizer"
        }
      }
    ],
    "type": "exception",
    "reason": "java.lang.IllegalArgumentException: Analyzer [custom_standard_lowercase_analyzer] not found.",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Analyzer [custom_standard_lowercase_analyzer] not found."
    },
    "header": {
      "processor_type": "text_tokenizer"
    }
  },
  "status": 500
}

```

---

<div class="post-metadata">

**Author:** ![purefield](https://avatars.discourse-cdn.com/v4/letter/p/57b2e6/32.png) [@purefield](https://discuss.elastic.co/u/purefield)\
**Post date:** [March 16, 2018, 9:16pm UTC](https://discuss.elastic.co/t/ingest-processor-plugin-with-analyzer-support/123365/5 "2018-03-16T21:16:41Z")

</div>

We solved the problem by defining the custom analyzer functionality directly using Lucene Classes.

Happy to share some code but unable to upload or include.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 13, 2018, 9:16pm UTC](https://discuss.elastic.co/t/ingest-processor-plugin-with-analyzer-support/123365/6 "2018-04-13T21:16:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.