# Elasticsearch Ingest Pipeline + index for language identification and text analysis

**URL:** <https://discuss.elastic.co/t/elasticsearch-ingest-pipeline-index-for-language-identification-and-text-analysis/241646>\
**Category:** Elasticsearch\
**Created:** [July 17, 2020, 1:17pm UTC](https://discuss.elastic.co/t/elasticsearch-ingest-pipeline-index-for-language-identification-and-text-analysis/241646 "2020-07-17T13:17:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![yolo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yolo/32/51497_2.png) [@yolo](https://discuss.elastic.co/u/yolo)\
**Post date:** [July 17, 2020, 1:17pm UTC](https://discuss.elastic.co/t/elasticsearch-ingest-pipeline-index-for-language-identification-and-text-analysis/241646/1 "2020-07-17T13:17:55Z")

</div>

Hi,

i am using elasticsearch ingest pipeline for language identification. Furthermore, i would like to apply to each language-field a language analyzer. To my knowledge there is no ingest pipeline for langauge analyzers so i created an index that will use my pipeline and apply the language analyzer from the mapping.

here is my index:

```
PUT my_index
{
  "settings": {
    "index.default_pipeline" : "my_sample_pipeline",
    "analysis" : {
      "analyzer": {
        "my_analyzer" : {
          "type" : "custom",
          "tokenizer" : "standard",
          "filter" : "my_apostrophe"
          }
      },
      "filter" : {
        "my_apostrophe" : {
          "type" : "asciifolding",
          "perserve_original": true
        }
      }
    }
  },  
  "mappings": {
    "dynamic": true,
    "properties": {
      "description": {
        "analyzer" : "my_analyzer",
        "type" : "text",
        "fields" : {
          "en_analyzer": {
            "type": "text",
            "analyzer": "english"
          },
          "de_analyzer": {
            "type": "text",
            "analyzer": "simple"
          },
          "pt_analyzer": {
            "type": "text",
            "analyzer": "portuguese"
          },
          "fr_analyzer": {
            "type": "text",
            "analyzer": "french"
          },
          "zh_analyzer": {
            "type": "text",
            "analyzer": "smartcn"
          }
        }
      }
    }
  }
}

```

and this is my pipeline:

```
// PUT _ingest/pipeline/my_sample_pipeline
{
  "processors" : [
    {
      "inference" : {
        "model_id" : "lang_ident_model_1",
        "inference_config": {
          "classification" : {
            "num_top_classes" : 1
          }
        },
        "field_map" : {
          "description" : "text"
        },
        "target_field" : "_ml.lang_ident"
        }
      },
      {
      "rename" : {
        "field" : "description",
        "target_field" : "description.raw"
        }
      },
      {
      "rename" : {
        "field" : "_ml.lang_ident.predicted_value",
        "target_field": "description.language_processed"
        }
      },
      {
      "script" : {
        "lang" : "painless",
        "source" : "ctx.description.supported = (['de', 'en', 'fr', 'pt', 'zh'].contains(ctx.description.language_processed))"
      }
      },
      {
      "set" : {
        "if" : "ctx.description.supported",
        "field": "description.{{description.language_processed}}",
        "value" : "{{description.raw}}",
        "override" : false
      }
      },
      {
      "set": {
        "if" : "ctx.description.language_processed == 'en'",
        "field" : "description.{{description.language_processed}}",
        "value" : "{{description.en_analyzer}"
      }
    }
  ]
}

```

after storing the description in its language-field using SCRIPT processor i am trying with the last SET processor to apply "description.en\_analyzer" on the description i stored in en\_field. Is this way of proceeding possible? the output of the last SET processor is an empty string field. Since the field en\_analyzer in the index mapping is empty. my target is to apply to my "classified descriptions" the appropriate analyzers. Any ideas how to proceed?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 14, 2020, 1:18pm UTC](https://discuss.elastic.co/t/elasticsearch-ingest-pipeline-index-for-language-identification-and-text-analysis/241646/2 "2020-08-14T13:18:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.