# Ingest pipeline for text analysis?

**URL:** <https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857>\
**Category:** Elasticsearch\
**Created:** [July 20, 2020, 8:44am UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857 "2020-07-20T08:44:23Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![yolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yolo/32/51497_2.png) [@yolo](https://discuss.elastic.co/u/yolo)\
**Post date:** [July 20, 2020, 8:44am UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/1 "2020-07-20T08:44:23Z")

</div>

Hi community,

is it possible to apply text analysis on search index using ingest pipelines? i still did not find any processor suitable for this task. The idea is to detect the language using inference processor from ingest pipeline and than apply language specific analyzers on the text. Is this even possible with elasticsearch?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 20, 2020, 9:51am UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/2 "2020-07-20T09:51:32Z")

</div>

Have a look at [https://www.elastic.co/blog/multilingual-search-using-language-identification-in-elasticsearch](https://www.elastic.co/blog/multilingual-search-using-language-identification-in-elasticsearch)

---

<div class="post-metadata">

**Author:** ![yolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yolo/32/51497_2.png) [@yolo](https://discuss.elastic.co/u/yolo)\
**Post date:** [July 20, 2020, 12:19pm UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/3 "2020-07-20T12:19:38Z")

</div>

Yes i have seen it. But still i am not sure if it is possible after language identification to use built-in language for each field. This is somehow not working. Not even in the example on that website. I used a serie of processors to store the input field in language specific field.  
I am now struggling with how to apply language analyzers to those fields. Is it even possible in ES?

- The language fields are difined in the pipeline and the language analyzers in the mapping.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 20, 2020, 12:37pm UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/4 "2020-07-20T12:37:04Z")

</div>

Isn't what " Per-Field" section is about?

---

<div class="post-metadata">

**Author:** ![yolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yolo/32/51497_2.png) [@yolo](https://discuss.elastic.co/u/yolo)\
**Post date:** [July 21, 2020, 10:15am UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/5 "2020-07-21T10:15:01Z")

</div>

i used the igest pipeline processors to create language per-field. Now i would like to apply language analyzers on the fields. But it somehow not working. the text is not analyzed.

my index:

```
{
  "settings": {
    "index.default_pipeline" : "my_sample_pipeline",
    "analysis" : {
      "analyzer": {
        "my_analyzer" : {
          "type" : "custom",
          "tokenizer" : "standard"
             }
          }
      }
    }
  },  
  "mappings": {
    "dynamic": true,
    "properties": {
      "description": {
        "analyzer" : "my_analyzer",
        "type" : "text",
        "properties" : {
          "en_analyzer": {
            "type": "text",
            "analyzer": "english"
          },
          "de_analyzer": {
            "type": "text",
            "analyzer": "simple"
          },
          "fr_analyzer": {
            "type": "text",
            "analyzer": "french"
          }
        }
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 22, 2020, 4:01am UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/6 "2020-07-22T04:01:06Z")

</div>

> [@yolo](#):
>
> the text is not analyzed.

How do you know?

---

<div class="post-metadata">

**Author:** ![yolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yolo/32/51497_2.png) [@yolo](https://discuss.elastic.co/u/yolo)\
**Post date:** [July 23, 2020, 8:15am UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/7 "2020-07-23T08:15:40Z")

</div>

i know this from my pipeline. I used second SET processor to assign to each language-per field a language analyzer . But i am not sure if i can use my analyzers in the pipeline.

My pipeline:

```
PUT _ingest/pipeline/my_sample_pipeline
{
  "processors" : [
    {
      "inference" : {
        "model_id" : "lang_ident_model_1",
        "inference_config": {
          "classification" : {
            "num_top_classes" : 1
          }
        },
        "field_map" : {
          "description" : "text"
        },
        "target_field" : "_ml.lang_ident"
        }
      },
      {
      "rename" : {
        "field" : "description",
        "target_field" : "description.raw"
        }
      },
      {
      "rename" : {
        "field" : "_ml.lang_ident.predicted_value",
        "target_field": "description.language_processed"
        }
      },
      {
      "script" : {
        "lang" : "painless",
        "source" : "ctx.description.supported = (['de', 'en', 'fr', 'pt', 'zh'].contains(ctx.description.language_processed))"
      }
      },
      {
      "set" : {
        "if" : "ctx.description.supported",
        "field": "description.{{description.language_processed}}",
        "value" : "{{description.raw}}",
        "override" : false
      }  
      },
      {
      "set" : {
        "if" : "ctx.description.language == 'en'"
        "field": "description.{{description.language_processed",
        "value" : "{{description.en_analyzer}}"
      }
    }
  ]
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 23, 2020, 4:12pm UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/8 "2020-07-23T16:12:04Z")

</div>

Once the field has been set, the right analyzer for that field will be used. I don't understand what you're taking about.

---

<div class="post-metadata">

**Author:** ![yolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yolo/32/51497_2.png) [@yolo](https://discuss.elastic.co/u/yolo)\
**Post date:** [July 23, 2020, 8:23pm UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/9 "2020-07-23T20:23:57Z")

</div>

is there any possibility to store the analyzed text in a field and see how does the text look like after the pre-processing? (i know about Analyze API) but i would like to see it for the whole data. I was thinking using ingest pipeline allows me this .

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 23, 2020, 9:26pm UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/10 "2020-07-23T21:26:54Z")

</div>

No. The analyze API is built for that need.

---

<div class="post-metadata">

**Author:** ![yolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yolo/32/51497_2.png) [@yolo](https://discuss.elastic.co/u/yolo)\
**Post date:** [July 23, 2020, 9:52pm UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/11 "2020-07-23T21:52:32Z")

</div>

when using language analyzers in index mapping , are the analyzers applied to the indexed text / indexing time (my datasets) or to query text /search time? Thank youuuu

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [July 23, 2020, 10:07pm UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/12 "2020-07-23T22:07:19Z")

</div>

Both.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 20, 2020, 10:07pm UTC](https://discuss.elastic.co/t/ingest-pipeline-for-text-analysis/241857/13 "2020-08-20T22:07:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.